Skip to content

PATTERN Cited by 1 source

Async agent invocation over Kafka

Pattern

Put a Kafka-style broker (Amazon MSK, Redpanda, self-hosted Kafka) between the caller and an agent-orchestration runtime so that:

  • Caller publishes a request to an inbound topic and moves on.
  • A consumer (Lambda, long-running worker, another agent) pulls the event and invokes the agent runtime asynchronously — the invocation does not block the consumer on agent completion.
  • Agent result is later published to an outbound topic; downstream consumers process the decision / fraud alert / case escalation on their own clock.

Canonical instance: IBM + AWS KYC architecture on AWS. "Lambda functions serve as the integration layer, consuming events from MSK, invoking AgentCore asynchronously, and publishing results back to Kafka topics for downstream system consumption." (Source: sources/2026-04-23-aws-modernizing-kyc-with-aws-serverless-solutions-and-agentic-ai.)

Why async invocation specifically

Three properties make an async boundary between Kafka and an agent runtime load-bearing at scale:

  1. Consumer throughput is decoupled from agent latency. A synchronous invoke would tie the Kafka consumer to the agent's think-time (seconds to minutes in a multi-sub-agent case). Async invoke lets the consumer drain Kafka at Lambda speed and let agent invocations accumulate in the runtime's async queue.
  2. Retry + DLQ lives on Kafka. If the agent fails, Lambda can re-publish to a retry topic or route to a DLQ without holding open the original consumer offset. Standard Kafka consumer-group semantics apply.
  3. Fan-out scales independently. The Supervisor dispatches to five sub-agents; those dispatches don't back up the MSK consumer. This is a prerequisite for the IBM + AWS claim of "thousands of concurrent KYC requests" at sub-5-minute latency.

Diagram

 producer ─► MSK inbound topic ─► Lambda consumer ─► AgentCore.invokeAsync()
                                                         │
                                                         ▼
                                                    AgentCore runtime
                                                         │
                                                         ▼
                                               (Supervisor + sub-agents)
                                                         │
                                                         ▼
                                               publish result
                                                         │
                                                         ▼
                                        MSK outbound topic ─► downstream

Note that Lambda consumer returns immediately after invokeAsync — it does not wait for the result. The result path is entirely topic-to-topic.

Requirements on the agent runtime

To make this pattern viable, the agent runtime has to:

  • Accept async invoke. systems/bedrock-agentcore's "MSK consumers trigger AgentCore processing without blocking" is the explicit disclosure.
  • Preserve context across async hops. A KYC case may involve several async steps (Supervisor dispatch → sub-agent run → additional verification → final composition). Context has to survive those hops. That's what systems/agentcore-memory is for.
  • Publish results downstream on completion. The runtime has to emit a completion event the Lambda (or a direct Kafka producer inside the agent) can route back to the outbound topic.

When to reach for it

  • Agent work is seconds-to-minutes latency (too long to keep a sync path open).
  • High ingest throughput with lumpy agent compute — bursty KYC application waves, event-driven fraud spikes.
  • Clean failure semantics matter — retry / DLQ / reprocessing is way easier on Kafka than on a sync API.

Anti-patterns

  • Sync invoke through a Kafka consumer. If the Lambda blocks on invokeSync, you've lost the async benefit — congestion at the agent runtime now backs up Kafka consumption, which backs up producers.
  • Agent-result write through the same async boundary as input. Racing a completion-event write into the inbound topic risks loop + replay bugs. Use separate inbound and outbound topics (patterns/inbound-outbound-topic-pairing).
  • Correlation-id amnesia. Producer → consumer → agent → outbound publish must preserve a correlation id so downstream consumers can join decisions back to the original request. The KYC post implies but doesn't explicitly name this; it is foundational in any real implementation.

Variant: the streaming-native agent runtime

If the agent runtime natively speaks Kafka (emerging runtimes like Redpanda Agentic Data Plane — see patterns/multi-agent-streaming-coordination), the Lambda consumer layer collapses: agents subscribe to the inbound topic directly. The KYC post uses Lambda because AgentCore doesn't natively speak Kafka — that's a current-state limitation, not a requirement of the pattern.

Seen in

Last updated · 476 distilled / 1,218 read