Skip to content

REDPANDA 2025-10-28

Read original ↗

Redpanda — Introducing the Agentic Data Plane

Summary

Alex Gallego's (Redpanda founder/CEO) productization follow-up to his 2025-04-03 autonomy essay, naming the commercial shape of that vision: the Agentic Data Plane (ADP) — a "unified runtime and control plane that safely exposes enterprise data to AI agents". Where the April essay canonicalised enterprise autonomy as code-in- control of end-to-end agent flow with MCP as the integration layer, this 2025-10-28 post names the four architectural subsystems ADP composes and announces Redpanda's acquisition of Oxla — a C++ distributed query engine with PostgreSQL wire protocol and separated compute-storage — as the context-management substrate. The load-bearing framing shift: Gallego reframes CIO fear of agents from "will the agent hallucinate?" to "will the agent access the wrong data?" and "when something goes wrong, can I explain it?" — then argues governance (access controls + observability) is the primary product surface, not a bolt-on feature. Tier-3 borderline include: marketing-heavy launch post with zero production numbers; passes on architectural-vocabulary grounds for three canonicalisations the wiki lacked (ADP-as-product-shape, Oxla-as-system, governed-agent-data-access concept + OBO authorization pattern).

Key takeaways

  1. The Agentic Data Plane is a four-layer composition. Verbatim: "The ADP is a unified runtime and control plane that safely exposes enterprise data to AI agents. It combines: (A) a low- latency streaming layer for events and HITL workflows, (B) a distributed, Iceberg-native query engine for real-time context, (C) 300+ high-quality connectors to bring context to the models, and (D) a fully managed, global policy and observability layer that enforces access, records intent, and enables replayable audits." Canonicalised on the wiki as systems/redpanda-agentic-data-plane — a product-tier page naming the composition of pre-existing Redpanda streaming + Oxla query engine + systems/redpanda-connect 300+ connectors + a new global policy/observability layer.

  2. Oxla acquisition. Verbatim: "Redpanda has acquired Oxla: a team obsessed with performance, correctness, and data catalogs. Oxla (rpk oxla) is a distributed query engine, single binary, written in C++, built for demanding Iceberg queries and merging real-time context with historical data. It is a PostgreSQL wire protocol engine with separated compute-storage, oriented to bring low-latency context management for agents looking to merge streams or large data sets, search, or simply filter aggregations in real time." First wiki disclosure of Oxla as a system; early preview mid-December 2025. Canonical framing: "As agents merge unbounded historical queries, possibly accessing petabytes of data, SQL is the best mechanism to filter and aggregate while the model summarizes." — positions SQL-over-Iceberg as the filter-and-aggregate primitive beneath LLM summarisation.

  3. Governance is the product, not the feature. Verbatim: "The fear from CIOs is not the code of the agent itself, it is governance. In simple terms, it is access controls: can I trust that data is accessed by the right things? And observability: when things go wrong, can I understand what happened?" Canonicalised on the wiki as concepts/governed-agent-data-access — two-axis framing (access controls + observability) as the primary CIO-facing design surface for enterprise agents, distinct from hallucination-mitigation / quality-axis framings common in consumer-agent product marketing. Concrete substrate listed: "OBO to task-based authentication, DLP hooks, per-agent consent workflows, and immutable audit trails with configurable retention."

  4. Remote MCP + on-behalf-of (OBO) authorization with IdP integration. Verbatim item in the "Things we've built" list: "Remote MCP + authentication + authorization for OBO (on-behalf- of) workloads with IdP integration." Canonicalised as patterns/on-behalf-of-agent-authorization — the access-control pattern where the MCP server proxies a tool call on behalf of the authenticated human user (or calling service) through to the downstream data system, carrying the user's identity + scoped consent rather than a shared agent-service token. Structural distinction from "API era of root-token permissions, with all-or-nothing as the norm" verbatim — the legacy failure mode ADP is pitched against.

  5. Agent Runtime + knowledge-agent templates for common enterprise data sources. Listed in "Things we've built": "A set of knowledge-based agent templates for common data sources, including Git for code repos, Jira, GDrive, etc." plus "A declarative Agent Runtime." Product-tier extension of Redpanda Agents SDK (2025-04-03) — moving from library + SDK to opinionated agent templates and a declarative runtime.

  6. Redpanda Streaming remains the HITL + observability + replayable-audit substrate. Verbatim: "Redpanda's real-time streaming engine gives us a foundational layer for Human-in-the- Loop (HITL), async mailboxes, durable model replay, and observability." The streaming log (the original Redpanda product) is the substrate beneath ADP — canonical continuity with durable-execution framing from Gallego's April essay.

  7. Three-shift architectural narrative. Gallego's historical arc in the post: "The categorical shifts in data started with moving structured, self-hosted data to the cloud. Separating compute and storage ... The second shift was the idea that you can do analytics on structured and unstructured data with the same tools, which gave rise to the lakehouse — a shapeless dumping ground, leaving the complexities of managing the data to the query engines. The third shift is now here with AI Agents. AI has outgrown the static nature of warehouses and lakehouses into live operational and analytical data. Agents don't just need to watch data flow by. They need to reach into it, interact with it, and act on it safely." Canonical three-shift framing: compute-storage separation → lakehouse → agentic data plane. Positions ADP as the next structural category beyond the lakehouse.

  8. Open protocols, zero lock-in commitment. Verbatim: "The commitment is: open protocols, zero lock-in, because you can't afford it; MCP, A2A, for agents; PostgreSQL to filter in SQL, summarize in model; durable log for HITL; and Iceberg for long- term data state. The best tool for the job, either self-hosted or in any cloud." Named protocols: MCP (model context), A2A (agent-to-agent), PostgreSQL wire protocol (Oxla inherits this), durable log (Kafka wire protocol on Redpanda), Iceberg (table format).

Systems named

Concepts extracted

Patterns extracted

Operational numbers

  • 300+ connectors (same catalog as Redpanda Connect).
  • Early preview: mid-December 2025 for Oxla integration.
  • Zero production numbers disclosed: no latency, no throughput, no fleet counts, no customer-workload benchmarks. Pure architectural-category-naming post.

Caveats

  • Launch-post / marketing voice. Written by the founder; published the same day as the Oxla acquisition. "Today marks a singular moment in time for me and Redpanda" framing sets the tone. Architectural content is real but buried under vision prose.
  • Zero quantitative disclosures. No p99 latency for Oxla against Iceberg workloads; no agent-tool-call throughput; no customer reference architectures disclosed.
  • Oxla technical depth is thin. Post names four properties (C++, single binary, PG wire, separated compute-storage) and three workload targets (merge streams + large data sets, search, filter aggregations in real time) — the actual query engine architecture (planner, executor, catalog model, concurrency model, scheduling) is not disclosed. Follow-up architecture post will be required for a system-altitude deep ingest of Oxla.
  • OBO + IdP integration disclosed as a product line item, not a mechanism. Concrete enforcement shape (JWT-vs-mesh-auth, token-exchange flow, consent-scope model) is not disclosed; the pattern canonicalisation on the wiki is therefore structural (proxy carries caller identity) not mechanistic (which token flow, which consent vocabulary).
  • "300+ connectors" is the Redpanda Connect catalog (Benthos heritage), not new ADP-specific integrations. Reader should not count this as net-new architectural capability — it's a rebranding of the existing connector surface as ADP layer (C).
  • "Things we've bought" — Oxla is the only disclosed acquisition; "rolling integration" means the Oxla query engine is not yet integrated into the shipping Redpanda product at post publication.
  • "Things we're doubling down on" — governance framing names "OBO to task-based authentication, DLP hooks, per-agent consent workflows, and immutable audit trails with configurable retention" as the scope, but none of the four is disclosed at mechanism altitude. They are product-roadmap commitments.
  • A2A protocol named but not described. The "MCP, A2A, for agents" commitment references agent-to-agent as a protocol peer to MCP; neither the post nor prior Redpanda posts canonicalise A2A substrate on the wiki.
  • No competitive comparison. ADP is positioned against "API era of root-token permissions" and "warehouses and lakehouses [that] didn't have access to your specific data" — neither specific vendors named nor feature-by-feature contrasts drawn (Databricks Unity + AI Gateway, Snowflake Cortex + Agent features, AWS Bedrock Agents, etc. are ADP's architectural peers but not engaged).
  • Unsigned-but-self-attributed. Closes with "This note was handcrafted by a hooman. .alex" — Gallego-authored founder- voice piece, like the 2025-04-03 autonomy essay.
  • Tier-3 borderline include. Redpanda is Tier-3; post is launch-marketing with architectural content ~25-30% of the body. Passes borderline-case test on vocabulary-canonicalisation grounds (three new load-bearing wiki primitives: ADP-as- product-shape, Oxla-as-system, governed-agent-data-access concept + OBO pattern). Under the AGENTS.md filter a stricter disposition is defensible ("Only skip if architecture content is <20% of the body") — the call here is "include, but keep the ingest lean until a follow-up architecture post ships".

Cross-source continuity

  • Direct sequel to Gallego 2025-04-03 autonomy essay published 7 months earlier. April essay = vision + three substrate pillars (streaming + MCP + Python SDK); October post = product-tier packaging (ADP) + acquisition (Oxla) + governance productization (OBO + audit + consent).
  • Companion to sources/2025-06-24-redpanda-why-streaming-is-the-backbone-for-ai-native-data-platforms|2025-06-24 streaming-as-backbone essay — that post framed streaming as the data-substrate half of an AI-native data platform; this post extends into the governance
  • query-engine surfaces.
  • Extends Redpanda Agents SDK canonical coverage from April 2025 library + SDK framing into declarative Agent Runtime + knowledge-agent templates.
  • First wiki introduction of Oxla — C++ distributed query engine with PostgreSQL wire protocol + separated compute- storage + Iceberg-native. Deserves its own system page (thin at publication; richer after Q1 2026 technical disclosures).
  • Three-shift narrative brackets prior wiki coverage: concepts/compute-storage-separation canonicalised via Snowflake / Canva / Aurora ingests; concepts/data-lakehouse canonicalised via Redpanda 2025-01 medallion post; this post names the agentic-data-plane as a putative third category beyond them (wiki treats this as vendor-framing, not yet industry-canonical vocabulary).

Source

Last updated · 470 distilled / 1,213 read