Skip to content

SYSTEM Cited by 1 source

Amazon MSK (Managed Streaming for Apache Kafka)

What it is

Amazon MSK is AWS's managed Apache Kafka service. Customers get standard Kafka clients, topics, partitions, consumer groups, and exactly-once semantics (via Kafka Streams / transactions); AWS runs the brokers, ZooKeeper/KRaft, patching, storage, and inter-AZ replication. The Kafka API and wire protocol are unchanged — existing producers and consumers work as-is.

This page is a stub; MSK inherits almost all design-level content from systems/kafka. See that page for partitioning, replication, consumer-group, and compaction semantics.

Pattern of appearance

MSK shows up in cloud-native AWS architectures as the async backbone wherever the design calls for decoupled, durable event streaming between serverless / AI components and enterprise systems.

  • Agentic-AI orchestration backbone. In the IBM + AWS KYC architecture (2026-04-23) MSK is the communication substrate between the Supervisor Agent, five specialized sub-agents, and on-prem financial systems. Inbound topics carry KYC requests, document uploads, ID-verification vendor responses, and transaction events; outbound topics carry KYC decisions with confidence scores, human-review escalations, and fraud alerts. patterns/inbound-outbound-topic-pairing is the shape. (Source: sources/2026-04-23-aws-modernizing-kyc-with-aws-serverless-solutions-and-agentic-ai.)
  • Lambda as the integration shim. "Lambda functions serve as the integration layer, consuming events from MSK, invoking AgentCore asynchronously, and publishing results back to Kafka topics for downstream system consumption." MSK → Lambda → AgentCore → MSK is the canonical async-invocation loop. See patterns/async-agent-invocation-over-kafka.

Why MSK over Kinesis for this shape

The AWS canon has two streaming substrates: Kinesis (AWS-native proprietary protocol) and MSK (standard Kafka protocol). The KYC post's choice of MSK is load-bearing for two reasons:

  1. Kafka is the financial-services streaming default. Core- banking, AML, and case-management systems already speak Kafka; MSK lets the cloud layer integrate over the same protocol with on-prem brokers via MirrorMaker or standard replication.
  2. Consumer-group semantics give independent-rate draining per sub-agent, matching the "independent scaling of individual agents based on workload patterns" design goal.

Kinesis's shard model would impose different backpressure semantics (per-shard read quota, enhanced fan-out) not as naturally suited to a multi-agent fan-out topology.

Relationship to AgentCore

In the KYC architecture, MSK is outside the agent runtime boundary; AgentCore provides orchestration + session state + memory inside the agent cluster. MSK is the transport between requests and agent invocations and between agent results and downstream consumers — never a direct agent dependency. The async invocation boundary (MSK consumer → Lambda → AgentCore async invoke) is what makes the sub-5-minute KYC latency target compatible with "thousands of concurrent KYC requests".

Caveats

  • No internals disclosed. The KYC post names MSK in the architecture but discloses no broker count, partition count, throughput, or retention numbers. Treat this page as a hook until a sizeable MSK-internals post lands.
  • Not MSK Serverless vs provisioned. The post doesn't differentiate; both would satisfy the described architecture.

Seen in

Last updated · 476 distilled / 1,218 read