Skip to content

Redpanda — Redpanda 24.3 extends lakehouses with streaming data & CDC

Summary

Redpanda's 2024-12-03 24.3 release roundup is the beta announcement of six features that later became the canonical building blocks of the 2025 wiki coverage of Redpanda. It's a multi-headline launch post with moderate architecture density — vendor-launch voice throughout — but each of the six disclosed betas introduces a primitive the wiki has subsequently canonicalised under later-released source ingests. Specifically this is the origin-point disclosure for:

  1. Iceberg Topics beta on self-managed Enterprise + Redpanda Cloud BYOC — promoted to GA five months later in 25.1 (sources/2025-04-07-redpanda-251-iceberg-topics-now-generally-available|2025-04-07).
  2. Mountable Topics — safe zero-data-loss unmount/remount of unused Tiered-Storage topics via rpk CLI. First wiki source for this primitive. Canonicalised as concepts/mountable-tiered-storage-topic + patterns/hibernate-unused-topics-on-tiered-storage.
  3. Leader pinning — write-path locality dial on multi-region / multi-AZ clusters. Previously single-sourced to the 2025-02-11 HA post; this post is the original announcement and becomes the second citation on the concept page.
  4. Follower fetching framed as the complement to leader pinning — same "original announcement / earlier citation" relationship.
  5. Postgres CDC beta in Redpanda Connect — the postgres_cdc input, "optimized for Redpanda Connect's native Go (vs. Debezium's Java)", and "the beginning of a larger CDC effort". Previously single-sourced to the 2025-03-18 CDC tour post; this post is the first-engine canonical announcement that preceded it.
  6. rpk connect --secrets interpolation from external secrets managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, Redis). Canonicalised as patterns/external-secrets-manager-interpolation — first wiki source for this pattern.

Plus two supporting disclosures:

  • Redpanda Migrator offset translation — Migrator now supports per-consumer-group offset-translation map so consumers can fail over between source and target clusters without losing stream position. Canonicalised as concepts/cross-cluster-offset-translation-map — distinct from offset-preserving replication (the 2025-11-06 Shadowing property): Shadowing preserves offsets byte-for-byte so no map is needed; Migrator maintains an explicit translation map. The wiki's two canonical answers to the consumer-failover problem on Kafka-compatible substrates.
  • Customer-Managed VNets on Azure for Redpanda BYOC — customer manages the networking lifecycle of the VNet the data plane runs in, extending the prior customer-managed-VPC options on AWS + GCP.

Scope disposition. Tier-3 borderline include on origin-point-canonicalisation grounds. This post is a feature-announcement roundup — vendor-launch voice, no benchmarks, no production numbers, no architecture diagrams. But it's the earliest wiki-visible source for seven primitives the wiki has since canonicalised under later-release ingests (Iceberg Topics, Mountable Topics, leader pinning, follower fetching, postgres_cdc, external-secrets-interpolation, cross-cluster offset-translation). Two of those seven are net-new canonicalisations not previously on the wiki (Mountable Topics, secrets-manager interpolation); the rest become original-source backing for concepts the wiki currently cites from downstream elaborations. Architecture density ~25-30% on ~1,400-word body (Iceberg Topics subhead reinforces the pedagogy altitude; Mountable / Leader pinning / Follower fetching subheads carry real mechanism disclosure in 1–2 paragraphs each; Postgres CDC + secrets-manager carry real mechanism; Azure Marketplace, scheduled-maintenance, Terraform provider, 99.99% SLA, AI connectors are capability-statements only).

Source claim

On Iceberg Topics:

"Extend your data lake or cloud data warehouse with streaming data, delivered automatically and with zero ETL. Iceberg Topics in Redpanda unify streaming and analytics workloads, giving teams more choice for querying data and greater simplicity in how it flows into your data warehouse or data lake."

On Mountable Topics:

"'Hibernate' unused topics to conserve cluster resources and maximize cost savings and flexibility. You can now mount and unmount unused Tiered-Storage topics from a Redpanda cluster safely with zero data loss."

On leader pinning:

"For a Redpanda cluster deployed across multiple availability zones (AZs) or multiple regions, leader pinning ensures that topic-partition leaders are geographically closer to their producers, optimizing write latency and cost efficiency."

On Postgres CDC, framed as the native-Go implementation foil against Debezium/Java:

"The new PostgreSQL connector supports multiple replication modes and marks the beginning of a larger CDC effort that is optimized for Redpanda Connect's native Go (vs. Debezium's Java). The postgres_cdc input is an Enterprise-tier connector, available in beta for self-managed Redpanda Connect, and in Redpanda Cloud via Redpanda Connect."

On secrets-manager interpolation:

"Redpanda Connect now allows the rpk connect CLI flag --secrets to interpolate secrets values from external systems like AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and Redis. No more fumbling with different sets of environment variables or embedding credentials in your YAML!"

On Redpanda Migrator offset translation:

"Redpanda Migrator now supports offset translation for consumer applications that need to switch between reading data from the source cluster and the target cluster without losing their place in the stream."

Key takeaways

  1. Iceberg Topics were always BYOC-first. The beta ships for self-managed Redpanda Enterprise and Redpanda Cloud BYOC simultaneously — not Dedicated. This is the origin of the wiki's ongoing Iceberg-Topics-as-BYOC-data-ownership framing (canonicalised on concepts/byoc-data-ownership-for-iceberg) — the BYOC shape was the launch shape, not a follow-on extension.
  2. Iceberg Topics compose per-topic, not per-cluster. "The integration works on a per-topic basis, allowing you to mix and match Iceberg Topics alongside other regular topics in the same cluster." Establishes the per-topic-opt-in model later canonicalised as iceberg_enabled: true (concepts/iceberg-topic-mode axis) on the 2025-05-13 BYOC beta post.
  3. Mountable Topics = hibernation for unused Tiered-Storage topics. Extends tiered storage's fast-decommission property from the broker-lifecycle altitude to the topic-lifecycle altitude: a topic whose data is already in object storage can be unmounted (hidden from the cluster's active set) without losing data, then re-mounted to the same or a different cluster "even in a new cluster and with a new name." Uses the existing object-store substrate as its durability layer. Canonicalised as concepts/mountable-tiered-storage-topic. Operational pattern: patterns/hibernate-unused-topics-on-tiered-storage.
  4. rpk surfaces the lifecycle as explicit mount / unmount CLI verbs. Same verbs on the Redpanda Cloud API. First canonical wiki instance of Redpanda elevating storage-lifecycle primitives to first-class CLI verbs (precursor to the 2025-05-13 BYOC tutorial's topic-level iceberg_enabled configuration surface).
  5. Leader pinning + follower fetching framed as write-path/read-path duals from the original announcement. Verbatim: "Leader pinning complements follower fetching, which lowers costs for consumers with geographically- optimized reads." The dual framing is already canonicalised on the concept pages from the 2025-02-11 HA ingest; this post is the earliest source for it. Available on both self-managed Enterprise and Redpanda Cloud BYOC + Dedicated — the enterprise-licensing framing was present from launch.
  6. Postgres CDC explicitly native-Go-over-JVM-Debezium-Java. Canonical differentiation framing: "optimized for Redpanda Connect's native Go (vs. Debezium's Java)". Sits as first-engine origin of the Redpanda Connect CDC family that grew through 2025–2026 (MySQL next, then MongoDB / Spanner → MSSQL → Oracle). The post explicitly signals this trajectory: "marks the beginning of a larger CDC effort ... Stay tuned for our upcoming CDC connector for MySQL, which uses binary log file replication!"
  7. External secrets-manager interpolation eliminates YAML credential embedding. Four named backends (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, Redis) — same list that the 2026-04-09 Oracle CDC post will later canonicalise as the companion to Oracle Wallet's file-based credential store (sources/2026-04-09-redpanda-oracle-cdc-now-available-in-redpanda-connect). Canonicalised as patterns/external-secrets-manager-interpolation.
  8. Redpanda Migrator offset translation = the pre-Shadowing answer to cross-cluster consumer failover. Maintains a per-consumer-group offset-translation map so that a consumer switching between source and target clusters resumes at the correct offset. Canonicalised as concepts/cross-cluster-offset-translation-map. Load- bearing architectural foil against the 2025-11-06 Shadowing feature which removes the translation map as a DR critical-path dependency via byte-for-byte offset preservation. The wiki now has the full two-point history: Migrator with translation map (2024-12) → Shadowing with offset preservation (2025-11).
  9. Customer-Managed VNets on Azure complete the BYOC customer-network-lifecycle surface. Azure joins GCP and AWS with customer-managed private-network control — the operator manages the VNet lifecycle while the data plane still runs inside the customer's cloud account. Extends systems/redpanda-byoc's existing customer-managed-VPC options.
  10. Azure Marketplace launch + Terraform provider + 99.99% uptime SLA are capability-statement-altitude disclosures listed but not mechanism-walked. Redpanda Cloud Dedicated clusters are available via annual commitment with metered billing in Azure's East US, North Europe, and UK South regions; Terraform provider lands in public beta for managing clusters / topics / users / ACLs / networks / resource groups.

Systems and concepts extracted

Systems

Concepts

Patterns

Operational numbers

  • Azure Marketplace regions at launch: East US, North Europe, UK South. Dedicated clusters on annual commitment with metered billing; Serverless + pay-as-you-go "coming soon."
  • 99.99% uptime SLA claimed for multi-zone Redpanda Cloud BYOC + Dedicated clusters — "exceeds the SLAs of many cloud-provider-native Kafka services and brings Redpanda Cloud up to parity with other third-party streaming data providers."
  • No throughput / latency numbers for Iceberg Topics at beta.
  • No Mountable-Topic operation-latency numbers — unmount duration / remount recovery time undisclosed.
  • No leader-pinning / follower-fetching latency deltas — the feature is announced but no before/after measurement is shared.
  • No Postgres CDC benchmark — contrast the 2025-11-06 MSSQL CDC launch which disclosed ~40 MB/s vs ~14.5 MB/s for an unnamed alternative.
  • No secrets-manager interpolation overhead data — latency of a rpk connect startup under secret interpolation not quantified.
  • No Migrator offset-translation-map storage footprint — per-consumer-group map sizing / refresh cadence undisclosed.

Caveats / what this post does not cover

  • Vendor-launch voice throughout — this is a product-release roundup, not a retrospective. Every claim is a capability statement; no production case study, no customer telemetry, no mechanism walk.
  • Iceberg Topics beta scope: "beta, Iceberg Topics functionality is supported for non-production use only." The post explicitly gates the feature to non-production use at launch; operators must "talk to your Redpanda customer success manager to enable access in a BYOC or self-managed environment." The GA follow-up in 25.1 (2025-04-07) removes this gate.
  • Mountable Topics operational surface underspecified. What happens to consumer group offsets on a topic that's unmounted then re-mounted to a new cluster with a new name? What's the failure-mode of a partial unmount? Retention behaviour during the unmounted state? These are deferred to product documentation.
  • Leader pinning mechanism not walked. The announcement is capability-statement altitude — "preferred locations for topic partition leaders" — without mechanism disclosure of how the preference is expressed, how Raft leader-election honours it, or what happens under failure of the preferred region. The 2025-02-11 HA post adds mechanism depth.
  • Follower fetching is a one-sentence bullet — no client.rack / KIP-392 mechanism disclosure. The 2025-02-11 post adds mechanism depth.
  • Postgres CDC multi-replication-mode framing ("supports multiple replication modes") is named but modes aren't enumerated (logical vs physical vs something else). The 2025-03-18 CDC tour post clarifies this as Postgres logical replication + replication-slot mechanics.
  • "Native Go vs. Debezium's Java" framing is asserted as an advantage without a quantitative benchmark. The claim later appears across the Redpanda Connect CDC posts (2025-03-18, 2025-11-06, 2026-04-09) — still without an apples-to-apples comparison against a Debezium Postgres workload.
  • Secrets-manager interpolation is CLI-only at launch. The "fully managed version of secrets management (coming soon) built into the console for all Redpanda Cloud deployments" is previewed but not shipped in this release. No post-launch wiki source re-visits the console flavour.
  • Redpanda Migrator offset translation mechanism — the translation-map storage substrate (where does the map live? how is it refreshed? what are its failure modes?) isn't walked. Contrast Shadowing's later offset preservation which removes the map entirely.
  • Scheduled maintenance windows, Terraform provider, 99.99% SLA are listed as bullet capabilities with zero mechanism or operational-experience narrative.
  • No Redpanda Operator integration for 24.3 features — the 2025-05-06 K8s guide later canonicalises the K8s path; this post doesn't name K8s.
  • Customer-Managed VNets on Azure disclosed as a link-out without mechanism depth. "With a standard BYOC cluster, Redpanda manages the networking lifecycle. For additional security, you can deploy the Redpanda data plane into your existing virtual network (VNet) and manage the lifecycle yourself."
  • AI connectors (Cohere, Bedrock, Vertex AI embeddings) + Bloblang Playground are single-paragraph capability statements. Downstream 2025–2026 AI substrate ingests canonicalise the AI-native-data-platform framing; this post is three months ahead of the Gallego autonomy essay ( 2025-04-03) that articulates the thesis.
  • Ockam + Timeplus partner connectors listed as Community-tier open-source — no mechanism disclosure; out-of-scope for this wiki's altitude.

Scope disposition

Tier-3 borderline include on origin-point-canonicalisation grounds. Release-roundup voice with vendor-launch altitude on every section, but six of the eight disclosed betas become wiki-load-bearing primitives under later-release ingests. Passes on:

  • Net-new canonicalisations (2): Mountable Topics as a topic-level tiered-storage hibernation primitive; external- secrets-manager interpolation for rpk connect.
  • Origin-point sources (5): Iceberg Topics, leader pinning, follower fetching, Postgres CDC (as first engine in the family), Redpanda Migrator offset translation. Each is currently wiki-cited from downstream elaborations; this post is the announcement that preceded them.
  • Architectural foil for 2025-11-06 Shadowing. The Migrator offset-translation-map primitive is the wiki's canonical counter-example to Shadowing's offset-preservation property, completing the two-point history of the Redpanda cross-cluster-consumer-failover axis.

Fails on throughput / latency / customer-case-study / benchmark grounds — zero production numbers beyond the capability-level 99.99% SLA claim.

Source

Last updated · 550 distilled / 1,221 read