Skip to content

AWS 2026-09-29

Read original ↗

Build adaptive AI interfaces with the AG-UI protocol, agent swarms, and Nova Act on AWS

Summary

An AWS Architecture Blog reference architecture for building interfaces that adapt at runtime to what an AI discovers, rather than to a layout fixed at design time. The motivating problem: generative-AI outputs are variable (one medical scan shows a single fracture, another twenty ambiguous regions), so a static UI is either over-built for the simple case or under-built for the complex one. The solution composes three technologies — the AG-UI protocol (a standard set of typed events streamed over Server-Sent Events so any agent framework can drive any frontend without bespoke integration code), the Strands Agents SDK Swarm pattern (peer agents that share hypotheses and debate to a confidence-scored consensus, with the debate visible in real time for explainability), and Amazon Nova Act (natural-language browser automation that submits validated findings into legacy systems that lack APIs). The worked example is a HIPAA-scoped radiology assistant on Bedrock AgentCore, where a React AG-UI client streams over SSE to an orchestrator that fronts a three-agent swarm (Image Analysis, Clinical Reasoning, Reporting), Nova Act writes to a legacy RIS/EMR, and every layer is designed around explainability, PHI safeguards, and human validation gates. It is a reference/best-practices post, not a production retrospective — the numbers (80% consensus threshold, 5 max debate rounds) are configuration defaults, not measured outcomes.

Key takeaways

  1. The problem is a mismatch between dynamic AI discovery and static UI design. "AI agents discover and reason about the world dynamically, while traditional interface design assumes static, predetermined outputs." Domains where AI discovers things (fraud evidence chains, legal clauses, security threat patterns, radiology regions of interest) rather than classifies into fixed categories hit this wall — you spend weeks building interface variations and maintaining multiple code paths as models evolve.

  2. AG-UI is a universal agent-to-UI contract over SSE. Before AG-UI, connecting an agent to a frontend meant custom WebSocket formats, polling, and bespoke glue re-written for every framework. AG-UI defines typed events streamed over Server-Sent Events — TEXT_MESSAGE_CONTENT (token-by-token agent reasoning), STATE_DELTA (incremental bidirectional state sync), TOOL_CALL_START / TOOL_CALL_END (tool execution visibility), and UI_COMPONENT_SPEC (agent tells the frontend which component to render and how). Agent frameworks emit events; frontends consume them — swap frameworks without rewriting integration code. (Source: this article)

  3. The Strands Swarm pattern is peer debate to consensus, not supervisor/worker delegation. Three specialized agents (Image Analysis, Clinical Reasoning, Reporting) operate as peers that share hypotheses and iterate through debate rounds until confidence scores clear a threshold (80% default) or max_rounds (5 default) is exhausted. The debate is the product: increasing confidence (72% → 78% → 85%) signals convergence; decreasing confidence (68% → 52% → 45%) is a false positive being successfully challenged; hitting max rounds without consensus marks a finding disputed. This is explicitly contrasted with two sibling multi-agent shapes (see Pattern selection below).

  4. Visible debate is the explainability mechanism. Rather than a black-box verdict, the interface streams each agent's reasoning character-by-character (color-coded per agent), a confidence timeline, and evidence cards with a Challenged / Consensus / Disputed status badge. Users "see why agents reached conclusions, not only what they concluded" — enabling error detection, confidence calibration, and radiologist education.

  5. Nova Act bridges legacy systems that have no API via natural-language browser automation. Hospital RIS/EMR systems often run on decades-old stacks that can't be modified. Nova Act drives their existing web UI with plain-language commands (nova.act("Click Sign In button"), nova.act(f"Search for patient MRN '{mrn}'")), retrieving credentials from Secrets Manager at runtime and streaming each action back to the UI as a live action log for transparency and intervention.

  6. The frontend sees one agent, not three — the orchestrator pattern. "Your frontend sees one agent (radiology-assistant), not three, through the orchestrator pattern." The orchestrator encapsulates swarm-coordination complexity behind a single entry point and coordinates the swarm by analysis stage. This simplifies integration.

  7. Agents pick from a predefined component library, not arbitrary HTML. Rather than generating arbitrary markup, agents select from themed, accessible components (ROICard, DebatePanel, ConfidenceMeter). This balances flexibility against design consistency and security — a constrained-output discipline for agent-driven UIs.

  8. Bidirectional state sync is what makes human-in-the-loop possible. The frontend exposes state (current findings, validation decisions) to agents via STATE_DELTA events; agents update state through actions. This synchronization "supports human-in-the-loop workflows where agents pause for validation before proceeding."

  9. PHI safeguards are load-bearing, not incidental. SSE-KMS with a customer managed key on all S3 buckets and DynamoDB tables; TLS 1.2+ enforced (bucket policy denies when aws:SecureTransport is false); S3 Block Public Access at account + bucket level; short-lived (300 s) pre-signed PUT URLs scoped to a per-user key prefix and application/dicom content type with an s3:content-length-range upload cap; CloudWatch Logs data-protection policies to mask PHI; Cognito JWT verified on the SSE endpoint before emitting any PHI-bearing event; type_text(..., sensitive=True) in Nova Act to keep credentials out of trajectory logs; a signed BAA covering all HIPAA-eligible services.

Architecture (16 components)

React app → CloudFront (serves the SPA from S3) → Cognito (JWT auth) → API Gateway (REST + SSE) → Lambda AG-UI handler → Bedrock AgentCore Gateway (orchestration/routing/session/load-balancing) → AG-UI handler (formats agent responses as AG-UI events) → AgentCore Runtime (isolated instance per agent type) → Strands Agent Swarm (Image Analysis + Clinical Reasoning + Reporting) → Bedrock (Claude Sonnet inference). Supporting: S3 (medical images, SSE-KMS), DynamoDB (session state, conversation history, findings, agent registry), OpenSearch Serverless (vector knowledge base for medical literature), Nova Act (legacy RIS/EMR browser automation), Secrets Manager (legacy credentials), CloudWatch + ADOT/X-Ray + AWS STS (observability + temporary credentials).

Key architectural decisions

  • Orchestrator pattern — frontend sees one agent (radiology-assistant); orchestrator internally coordinates the swarm by analysis stage.
  • Predefined component library — agents select ROICard / DebatePanel / ConfidenceMeter, not arbitrary HTML (flexibility + consistency + security).
  • SSE for real-time streaming — agent contributions render character by character; transparency into reasoning.
  • Bidirectional state synchronization — STATE_DELTA exposes frontend state to agents and agent actions back to the frontend; the substrate for pause-for-validation HITL.

Consensus mechanism (numbers)

  • consensus_threshold: float = 0.80 — all hypotheses must exceed 80% confidence.
  • max_rounds: int = 5 — debate caps at 5 rounds; unresolved findings marked disputed.
  • Worked example A (subtle fracture): R1 Image proposes ROI >80%, R2 Clinical validates, R3 all agree → consensus.
  • Worked example B (false positive): R1 >80%, R2 Clinical challenges as artifact (drops below 80%), R3–R4 confidence keeps dropping, R5 max rounds → disputed.

Pattern selection guidance

The post gives an explicit decision matrix (dynamic AG-UI vs static UI) and, importantly, distinguishes three multi-agent shapes:

  • Swarm (this article) — use when multiple perspectives improve accuracy (peer review, consensus), debate has value (explainability, error detection), there is no clear hierarchy (peers, not supervisor/worker), and iterative refinement helps. This is distinct from agents-as-tools where a supervisor delegates to specialists.
  • Agents as tools — clear task decomposition, independent subtasks, natural hierarchy (manager coordinating experts), no peer debate needed.
  • Sequential workflow — strict ordering (step B needs step A), checkpoints, well-defined stages, no benefit from parallel exploration.

Use cases beyond medical imaging

Fraud detection (1–20 variable evidence chains, multi-analyst perspectives, legacy banking systems), legal document review (unpredictable clause counts, multi-domain legal opinions, legacy case management), security event response (variable threat indicators, network/malware/threat-intel collaboration, legacy SIEM without APIs).

Anti-patterns

Avoid this approach for: simple classification (fixed categories), deterministic calculations (no uncertainty to debate), low-latency requirements (multi-round debate adds latency), cost-sensitive predictable outputs (swarm increases token usage), or minimal explainability needs.

Caveats

  • Reference/best-practices post, not a production retrospective. No throughput, latency, cost, or accuracy numbers; 80%/5-rounds are configuration defaults.
  • Code samples are educational — the post explicitly says to review all configs against your HIPAA Security Rule implementation before production.
  • Model ID is externalized deliberately. The example defaults to Claude Sonnet 4 but instructs reading the model ID from an environment variable / Parameter Store because Bedrock retires models on a lifecycle schedule — hardcoding risks failure at end-of-life.

Extracted systems / concepts / patterns

Source

Last updated · 766 distilled / 2,225 read