Skip to content

REDPANDA 2026-02-10

Read original ↗

Redpanda — How to safely deploy agentic AI in the enterprise

Summary

Blog recap of a talk by Tyler Akidau (Redpanda CTO, originator of the Google Dataflow / Apache Beam streaming model) at Dragonfly's Modern Data Infrastructure Summit titled "Deploying Agentic AI Scalably and Safely in the Modern Enterprise". Akidau's core thesis: the consumer-AI market is booming while enterprise agentic AI is stuck, because running agents on private data inside corporate networks requires solving a cluster of data-movement + governance + replay problems the streaming community already solved a decade ago. The post reframes eight named challenges of enterprise agentic AI as streaming use-cases and argues that six of the eight reduce to data-streaming problems (the other two — context querying and authentication — stay outside streaming's remit).

Introduces a rhetorical framing — the Dungeons & Dragons alignment chart — as a compact way to locate human workers (hired to sit in the lawful-good quadrant) vs. AI agents (at best chaotic-good, because "you don't know what you don't know" and there's no way to govern or audit them out of the chaotic column). The rhetorical move: without governance + auditing infrastructure, enterprises deploying agents are plugging chaotic-unknown workers into their private data + internet, and "what could possibly go wrong?".

Enumerates eight challenges for enterprise agent deployment and maps six to streaming primitives: (1) context building + maintenance = streaming ETL / CDC; (2) context querying = outside streaming; (3) authentication = outside streaming; (4) governance = enforcement at interconnection points / agentic data plane; (5) auditing = high-throughput, low-latency, durable logs; (6) replay + validation = classic streaming replay; (7) dynamic routing = use AI selectively, route to ML/heuristics otherwise; (8) multi-agent coordination = microservices-style decoupled messaging. The closing caveat: streaming is "a foundational piece" but not a complete solution — authN/authZ, multi-modal context catalog, querying, and durable workflow execution are still needed.

Tier-3 borderline include: the post is marketing-adjacent (pitches Redpanda's ADP at the close) and doesn't disclose mechanism depth for any single primitive, but its value is in the eight-challenge enumeration as a canonical checklist and the D&D-alignment rhetorical framing. Both were missing from prior wiki coverage of the ADP / autonomy / governance corpus.

Key takeaways

  1. Agent anatomy mirrors streaming platform anatomy. Akidau's simplified agent: receives input (from SaaS/DB), interacts with tools (e.g. MCP), produces output — "at a high level, an agent functions similarly to how a streaming platform operates". Canonicalised on the wiki via concepts/streaming-as-agile-data-platform-backbone — the structural claim that streaming is the substrate an AI-native data platform runs on, now extended to the agent-as-streaming- consumer altitude.

  2. AI agents are structurally chaotic, not lawful. Verbatim: "Agentic AI, on the other hand, mostly falls into the right column of the chart. Despite the guardrails and training that companies attempt to put AI through, the best outcome at this point is that of 'chaotic good' — because you don't know what you don't know. Without the ability to govern or audit an agent, you can't confirm the agent is doing exactly what it's supposed to do (and only what it's supposed to do)." Canonicalised as concepts/agent-dnd-alignment-framing — the rhetorical frame that agents default to the chaotic column and governance + auditing is the infrastructure that moves them leftward toward lawful.

  3. Eight challenges for scaling agentic AI safely. Verbatim enumeration: "context building and maintenance; context querying; authentication; governance; auditing; replay and validation; routing; multi-agent coordination." Six of eight are streaming problems. Canonicalised as concepts/agentic-ai-infrastructure-challenges — the eight-item checklist as a canonical wiki entry point.

  4. Context building + maintenance = streaming ETL/CDC. Verbatim: "Building and maintaining data for your agent is a classic streaming Extract, Transform, Load (ETL) use case. You want to create datasets that are useful for your agents, whether you're building a knowledge base that connects to a vector database like Pinecone or performing change data capture (CDC) and pulling that data into an Online Analytical Processing (OLAP) database for analytical queries." Structural claim: "The more you focus on keeping your data up to date, the more effective the agents will be" — feeds RAG + vector-DB freshness via CDC fan-out.

  5. Governance = enforcement at interconnection points, not at sources. Verbatim: "Enforcement at each data source is virtually impossible when your architecture includes datasets, vector and OLAP databases, SaaS tools, Kafka Streams, and beyond. To effectively govern a fleet of agents, focus on the interconnection points." And: "Enforcement in a single agentic data plane brings uniformity to the governance of technological sprawl. With streaming, you can enforce service-level objectives for latency, accuracy, and cost — and turn opaque agent behavior into governed workflows." Canonical link to the ADP + concepts/governed-agent-data-access framing from the 2025-10-28 launch pair — this post is the February 2026 reprise of that framing at talk-recap altitude.

  6. Auditing requires capturing full inputs + outputs, not just metadata. Verbatim: "Historically, we've chosen the more cost-effective option for auditing: logging metadata requests rather than entire bytes of data (i.e., User Y read Z number of bytes on such-and-such day). But with agents you need to be able to audit what the request was, and what the agent did in response to the request. You can't make inferences without having the full dataset." Streaming log as the substrate: "streaming systems are good at high throughput, low latency, and durable logs." Extends durable event log as audit envelope with Akidau's framing of metadata-only audit is insufficient for agents — the core novelty vs classical systems audit.

  7. Replay + validation = classic streaming replay. Verbatim: "Validating agent behavior is a classic streaming replay scenario. Audit logs can perform double duty to help you review and confirm whether the agent in question is actually doing the job you asked it to. You can record the agent's inputs and outputs, then reassess." Reinforces patterns/snapshot-replay-agent-evaluation at vision altitude.

  8. Dynamic routing: use AI only when other systems aren't enough. Worked fraud-detection example verbatim: "Take fraud detection, for example. Machine learning (ML) models and heuristics are cheaper and make more sense to scan most of your data (since fraud will likely only make up a small percentage). Once those systems identify an anomaly, a trained fraud detection agent can help you investigate further." Motivated by LLM cost + compute + latency: "LLMs have their uses, but they're not a fit for every problem — because they're also expensive, require a lot of compute, and aren't very fast." Canonicalised as patterns/dynamic-routing-llm-selective-use.

  9. Multi-agent coordination = streaming microservices. Verbatim: "Multi-agent coordination seems like another classic streaming use case. If you think about the microservices architecture, you get benefits like decoupled services, durability, and fan-in and fan-out inputs. Multi-agent scenarios also require scalable, decoupled communication. With streaming, you get easier maintenance and better durability for your multi-agent system." Canonicalised as patterns/multi-agent-streaming-coordination.

  10. Streaming is necessary but not sufficient. Closing verbatim caveat: "while streaming can help solve a lot of agentic AI challenges, it's not your answer for everything. You still need authN/authZ, a multi-modal catalog of contextual data (not just streaming data), querying, and a durable execution for workflows, among other things." This is the honest boundary of the streaming-as-agent-substrate framing.

Extracted entities

Systems mentioned - systems/redpanda — the substrate implicit throughout. - Agentic Data Plane (ADP) — named at the close as the "managed, governed data control plane that provides the missing layer companies need to safely and reliably integrate agentic AI". - MCP — named as the tool interface in Akidau's anatomy-of-an-agent diagram.

Concepts named - concepts/autonomy-enterprise-agents — the chaotic-unknown framing is the risk-side dual of Gallego's autonomy-as- capability framing. - concepts/governed-agent-data-access — ADP closing framing. - concepts/streaming-as-agile-data-platform-backbone — agent anatomy = streaming anatomy. - concepts/change-data-capture — context-building-and- maintenance mechanism. - concepts/retrieval-augmented-generation — knowledge-base freshness downstream of CDC. - concepts/audit-trail — metadata-only audit insufficient for agents.

New canonicalisations from this post - concepts/agent-dnd-alignment-framing — D&D lawful/chaotic × good/evil chart applied to agent governance. - concepts/agentic-ai-infrastructure-challenges — eight-item checklist of enterprise agent-deployment challenges. - patterns/dynamic-routing-llm-selective-use — use AI where it wins, route to cheaper ML/heuristics otherwise. - patterns/multi-agent-streaming-coordination — streaming broker as the decoupled coordination substrate for multi-agent systems.

Patterns extended - patterns/cdc-fanout-single-stream-to-many-consumers — context-building-and-maintenance use case. - patterns/durable-event-log-as-agent-audit-envelope — metadata-only-audit-insufficient framing extends the canonical pattern. - patterns/snapshot-replay-agent-evaluation — validation-via- replay framing.

Operational numbers / concrete details

  • No production numbers. This is talk-recap voice, not retrospective.
  • No named customers or deployment sizes.
  • One worked framing example: fraud detection — ML/heuristics scan the ~99% normal-traffic majority, AI agent investigates flagged anomalies. No detection-rate or false-positive numbers given.
  • No architecture diagrams in the post body (the agent-anatomy image + D&D chart + ETL/governance/auditing/coordination diagrams are referenced by caption but not rendered in the scraped text).

Caveats

  • Talk-recap voice. Not a mechanism post; ADP-product position at close is marketing-adjacent.
  • Zero production numbers. No fleet sizes, no customer references, no latency / throughput / cost distributions.
  • Eight-challenge enumeration is editorial, not mechanism. The list is a conceptual decomposition; each axis has much deeper mechanism complexity the post doesn't unpack (governance policy engines, replay determinism for LLMs, multi-agent consensus semantics, etc.).
  • D&D alignment framing is rhetorical, not diagnostic. It compresses agent-governance concerns into a 2×2 chart that doesn't map cleanly to actual access-control, auditability, and replay axes (which are orthogonal, not a single lawful-chaotic spectrum).
  • "Classic streaming problems" framing over-claims. Context- building via streaming CDC is well-understood; governance + auditing + replay at the agent-interaction altitude are still active areas (see patterns/durable-event-log-as-agent-audit-envelope caveats on LLM replay determinism, exactly-once across tool chains, PII-in-prompts).
  • Fraud-detection example is hypothetical. No disclosed customer reference; the "trained fraud detection agent" substrate (prompt vs fine-tuned model, tool-call surface, cost model) is left unspecified.
  • Competitive positioning underspecified. The ADP close is pitch-voice; no engagement with Databricks Unity AI Gateway, AWS Bedrock Agents, Snowflake Cortex, or Anthropic's MCP governance proposals.
  • Watch-the-talk deferral. Most mechanism depth is deferred to the YouTube recording of Akidau's talk rather than unpacked in the blog.

Cross-source continuity

Source

Last updated · 470 distilled / 1,213 read