Skip to content

REDPANDA 2026-01-06

Read original ↗

Redpanda — Build a real-time lakehouse architecture with Redpanda and Databricks

Summary

Tech-talk recap post (unsigned Redpanda author; ~1,100 words) summarising a joint Redpanda/Databricks session "From Stream to Table: Building a real-time lakehouse architecture with Redpanda and Databricks" with speakers Matt Schumpert (Redpanda) and Jason Reed (Databricks, formerly on Netflix's data team). The post walks the historical arc that produced today's open lakehouse architecture — Apache Hadoop-era data lakes → governance sprawl → Apache Iceberg (Netflix-originated) → early file-based catalogs → Iceberg REST catalog standardisation → broker-native Iceberg Topics → integration with Databricks Unity Catalog — and reframes Redpanda's broker-native Iceberg primitive and Databricks' governance catalog as the two halves of a single real-time lakehouse substrate. Two load-bearing verbatim slogans canonicalise wiki-already-covered primitives at joint-vendor altitude: Schumpert — "The goal of this partnership is to remove the artificial line between real-time data and analytical data" and Redpanda unsigned — "the stream is the table" + "Streaming data is analytics-ready by default". Reed supplies the Netflix-as-origin-of-Iceberg disclosure with the architectural framing "Iceberg provides a foundation that looks and behaves like a warehouse table, while remaining open and cloud-native" and the integration claim "The data shows up already structured, already governed, and already queryable." Borderline Tier-3 include: architecture content ~50% of the body, zero production numbers, heavy marketing-link density (tech-talk promotion, Iceberg Topics use-case page, Unity Catalog product page, documentation cross-links). Passes scope on historical-framing + Netflix-origin-disclosure + joint-vendor- framing grounds rather than new vocabulary — every primitive named is already canonicalised on the wiki (Iceberg, REST catalog, Iceberg topic, Unity Catalog, streaming-broker-as- Bronze-sink pattern, broker-native catalog registration pattern all pre-exist). Zero net-new concepts / patterns / systems.

Key takeaways

  1. Historical arc of the open lakehouse (Hadoop-era problem → Iceberg solution). Verbatim pedagogy: "In the Apache Hadoop® era, data lakes had major advantages over traditional data warehouses. They enabled schema-on-read, flexible ELT workflows, and support for multi-structured data, all while significantly lowering costs with cloud object storage." followed by the governance-sprawl problem: "as adoption grew, so did complexity. Sprawl became a serious challenge, and multiple teams operating on the same datasets introduced issues around governance and reliability." Netflix named verbatim as the source: "these challenges ultimately led to the creation of Apache Iceberg—initially developed internally at Netflix and later open-sourced as an Apache project." Extends systems/apache-iceberg with the Netflix-origin disclosure at Databricks-speaker altitude (prior wiki coverage had the 2017 open-source date but not the Netflix provenance named by a former-Netflix-employee now at Databricks).

  2. Iceberg REST Catalog centralises governance at the catalog layer. Verbatim: "Early Iceberg catalogs were often implemented as collections of files stored directly in object storage. But with many users, workloads, and vendors creating and managing Iceberg tables across shared object storage, metadata sprawl and governance gaps were becoming dire." The REST catalog protocol is then framed as the "single control plane" with three verbatim responsibilities: "Managing permissions and access control. Coordinating concurrent reads and writes. Dynamically granting engines access to data at runtime." Extends concepts/iceberg-catalog-rest-sync with the governance-framing altitude (the prior canonical source at 2025-04-07 framed REST catalog sync as the transport mechanism; this post frames it as the governance endpoint from the catalog-consumer side). Interoperability emphasised verbatim: "Different platforms, written in different languages and running in different environments, can exchange metadata and enforce governance by speaking the same protocol."

  3. "The stream is the table" (canonical slogan for concepts/iceberg-topic + [[systems/redpanda-iceberg- topics|Redpanda Iceberg Topics]]). Full section header: "Redpanda Iceberg Topics: the stream is the table." Function verbatim: "Redpanda's Iceberg Topics allow you to store topic data in the cloud in the Iceberg open table format, so you can query real-time data while it's still streaming. This grants you instant analytics on the freshest data without the complexities of traditional ETL processes." Zero-ETL alternatives-displaced claim verbatim: "Before Iceberg Topics, making streaming data available in a lakehouse typically required significant manual effort. Teams either built custom ETL jobs using frameworks like Spark or deployed heavyweight connector architectures to move data from streaming platforms into analytical systems. To add insult to injury, these pipelines were often brittle and operationally expensive." Four verbatim wins from teams adopting the shape: "Lower infrastructure costs. Faster time-to-insight. Fewer human hours spent on pipeline maintenance. More free time to build valuable data products and AI applications."

  4. Unity Catalog as governance hub + Redpanda integrates via Iceberg REST API. Four verbatim governance-hub responsibilities: "Fine-grained access control. Consistent security across workloads. Metadata management and lineage. Easy discovery for downstream users." Reed's framing verbatim: "The data shows up already structured, already governed, and already queryable." Integration-mechanism disclosure verbatim: "Redpanda integrates directly with Unity Catalog using the Iceberg REST API. Through this integration, Redpanda registers Iceberg tables, manages schema updates, deletes tables when necessary, and handles the full lifecycle of the data." This is the canonical patterns/broker-native-iceberg-catalog-registration pattern's Unity-Catalog-specific instance — Redpanda owns table creation, snapshot registration, schema evolution, and table deletion against Unity, not just against Glue/Polaris/BigLake.

  5. Three-system labour division: Redpanda = real-time performance and reliability; Iceberg = open transactional table format optimised for analytics; Unity Catalog = governance + optimisation + federation + lifecycle. Verbatim: "Redpanda delivers real-time performance and reliability at scale. Iceberg provides an open, transactional table format optimized for analytics. Unity Catalog adds governance, optimization, federation, and lifecycle management across the entire system." The "dual citizenship" framing of the data verbatim: "the data becomes accessible to both worlds: The Apache Kafka® ecosystem continues to consume data in real time. The Iceberg ecosystem gains access to analytics-ready tables that can be queried by any Iceberg-compatible engine connected to Unity Catalog."

Systems named

Concepts extracted

Patterns extracted

Operational numbers

None disclosed. The post is purely qualitative. No throughput, no latency, no customer counts, no commit-interval target, no snapshot-expiry cadence, no fleet sizes.

Historical arc

Pre-Hadoop data-warehouse era → Hadoop + HDFS data-lake era (schema-on-read, ELT, cheap object storage; governance-weak) → Iceberg era (Netflix-originated, later open-sourced; table semantics atop immutable object storage) → file-based-catalog era (early Iceberg deployments embedded catalog in object storage; metadata sprawl and governance gaps) → REST catalog era (standardised HTTP API for catalog governance + interoperability across engines) → broker-native integration (Iceberg Topics make the stream = the table with zero ETL) → joint governance (Unity Catalog federates, optimises, and governs).

The arc collapses two prior wiki-separate historical tracks:

This post's framing is the first on the wiki that brackets the two tracks together as a single arc ending in the stream-is-the-table move.

Caveats

  • Tech-talk recap voice, not architectural retrospective. Pitched to the Redpanda blog audience; full session deferred to video ("The full talk is free to watch, but if you're more of a skimmer, this post covers the key moments").
  • Marketing-link density. Heavy cross-linking to Redpanda product pages (Iceberg Topics use-case page, documentation index) and Databricks product pages (Unity Catalog, the Databricks home). "Iceberg Topics are now generally available in Redpanda Cloud across AWS, GCP, and Azure, as well as in Self-Managed and BYOC deployments" is product- promotion voice, not architecture disclosure.
  • Zero production numbers — no throughput, no latency, no commit-interval distribution, no customer fleet sizes, no before/after quantitative wins on the migration from Airflow/Connect pipelines.
  • Pedagogy-level historical arc. The Hadoop-era framing is textbook; the file-catalog-to-REST-catalog evolution is framed as a clean progression without walking through the 2021-2023 JDBC catalog / Hive Metastore / Nessie / intermediate shapes.
  • Netflix-origin disclosure is at speaker attribution altitude only. Jason Reed's "having worked on the data team at Netflix" is the architectural-provenance claim; no Netflix engineering blog post is cited, no founding-team name is given, no internal pre-Iceberg system name is disclosed.
  • REST catalog responsibilities list is generic. Permissions / concurrency / credential-delegation are named in the same bullets the 2025-04-07 GA post covered; this post re-frames them in governance-vendor voice but adds no mechanism depth.
  • Unity Catalog responsibilities list is product-tour altitude. Fine-grained ACL / consistent security / metadata
  • lineage / discovery are named without mechanism (how does lineage get captured across a streaming-produced table? not disclosed).
  • "Open standards" framing. Iceberg REST catalog is standard; Unity Catalog's federation / optimisation / lifecycle-management functions are Databricks-proprietary extensions. Post doesn't draw the line.
  • "Dead letter tables" + "Partitioning, performance, and optimizing queries" + "Handling dirty data" all mentioned as topics the video covers but not unpacked in the blog post. These link to canonical wiki pages (patterns/dead-letter-queue-for-invalid-records, concepts/iceberg-file-based-catalog partitioning, systems/redpanda-iceberg-topics workload management) but the blog itself elides them.
  • Alex Gallego's "the goal is to remove the artificial line" sibling framing is absent — it's Matt Schumpert delivering the equivalent slogan on behalf of Redpanda here.
  • No comparison to Databricks' own Delta Lake format as an alternative open table format. Delta Sharing / Delta Live Tables / Delta Lake not engaged despite Databricks being a speaker — post is pure Iceberg-route framing.
  • No byline (Redpanda default attribution).

Cross-source continuity

Source

Last updated · 470 distilled / 1,213 read