Skip to content

SYSTEM Cited by 1 source

Amazon Nova Act

Amazon Nova Act is an AWS service for browser-based automation driven by natural-language commands. It lets an AI agent interact with systems through their existing web interface — navigating login screens, searching for records, filling form fields, uploading files, and capturing confirmation numbers — without those systems exposing an API. Each action can be streamed back to the calling interface so a human can observe (and intervene in) the automation. (Source: sources/2026-09-29-aws-build-adaptive-ai-interfaces-with-the-ag-ui-protocol-agent-s-ff0dbca9)

Why it exists

Many enterprise systems — particularly in healthcare — lack modern APIs. Hospital Radiology Information Systems (RIS) and Electronic Medical Record (EMR) systems often run on decades-old stacks that can't be modified without significant effort. Nova Act is the pragmatic bridge: instead of an integration project against a system with no integration surface, the agent drives the legacy system's own web UI the way a human would, using plain-language instructions.

Programming model

Nova Act exposes an imperative, natural-language API wrapped in a Workflow context and a NovaAct browser session:

from nova_act import NovaAct
from nova_act.types.workflow import Workflow

with Workflow(model_id="us.amazon.nova-act-v1:0",
              workflow_definition_name="radiology-submission") as workflow:
    with NovaAct(starting_page=f"{ris_url}/login", headless=True, workflow=workflow) as nova:
        nova.act("Click on the username input field")
        nova.type_text(creds["username"], sensitive=True)
        nova.act("Click Sign In button")
        nova.act(f"Search for patient MRN '{mrn}'")
        nova.act("Click New Report button")
        nova.act(f"Fill findings textarea with: {findings_text}")
        nova.act("Click Submit Report button")
        result = nova.act("Find and return the RPT- confirmation number")
  • nova.act("<natural language>") — a single UI action described in plain language; can also return data scraped from the page (e.g. a confirmation number).
  • nova.type_text(value, sensitive=True) — types text; sensitive=True keeps the value out of trajectory logs/screenshots.
  • Workflow(model_id=..., ...) — the model powering the automation (us.amazon.nova-act-v1:0 in the example) plus a named workflow definition.

Security model (load-bearing)

Nova Act captures prompts and screenshots as trajectory data — which is a data-protection hazard when the automation touches PHI/credentials:

  • Never interpolate credentials or PHI into act() commands. Use type_text(..., sensitive=True) so credentials aren't captured in logs and trajectories.
  • Credentials live in AWS Secrets Manager, retrieved at runtime, never exposed to the frontend.
  • In production, suppress trajectory capture entirely for authentication steps, or route trajectory storage to a KMS-encrypted, access-controlled bucket covered by the BAA.
  • Do not set ignore_https_errors=True; if the legacy system uses self-signed certs, add its CA to the runtime trust store.
  • Additional production controls: IP allowlisting, session-timeout enforcement, MFA where the legacy system supports it, and audit logging of every action.

Action streaming and human-in-the-loop

Each Nova Act command streams back to the calling UI (via the AG-UI tool-call events), so the interface can display a live action log — authentication, navigation, form filling, confirmation capture. This transparency lets the user understand what the automation is doing and intervene if issues arise — a human-in-the-loop property applied to a write action into a system of record.

Role in adaptive interfaces (2026-09-29)

Nova Act is the third of three technologies in AWS's adaptive-interface reference architecture (with AG-UI and the Strands Swarm). After the swarm reaches consensus on findings and a human validates them, Nova Act submits the validated report into the legacy hospital RIS/EMR by driving its web UI, then captures the RPT- confirmation number. It occupies the "integration with legacy systems lacking API access" slot: "Use when browser automation is the only viable integration path and action transparency matters."

Caveats

  • Single-source disclosure. Everything here comes from one AWS Architecture Blog reference architecture; Nova Act's internals (how it maps natural language to DOM actions, reliability/retry behavior on flaky pages, cost, latency, supported regions, failure modes) are not characterized. Code samples are explicitly educational.

Seen in

Last updated · 766 distilled / 2,225 read