Skip to content

AWS 2026-10-28

Read original ↗

AWS: Automating agentic AI Well-Architected reviews with the AWS Agentic AI Lens

Summary

Smarsh — a communications-archiving and compliance SaaS for 7,000+ organizations in regulated industries — must pass a Well-Architected review before any agentic solution reaches a customer. Its first multi-agent workload (a compliance-alert review system on Amazon Bedrock AgentCore runtime, gateway, registry, and identity) had to be reviewed against the AWS Well-Architected Agentic AI Lens: 150 best practices across 41 questions and six pillars. That lens existed only as documentation, not as a selectable lens in the Well-Architected Tool. Smarsh made the lens executable by packaging the review as a Claude agent skill with three parts: lens_data.json (all 150 best practices as structured data — the single source of truth), assessment-guide.md (a five-value status rubric, an evidence checklist, and honesty rules), and build_report.py (a deterministic openpyxl renderer that validates the model's JSON assessment and aborts on unknown IDs or invalid statuses). The model reasons and cites repository evidence; deterministic code renders and validates. Before trusting the skill, the team evaluated it against two synthetic fixture repositories: 16/16 expectations passed with the skill vs 8/16 without it. On the real workload, the skill produced an evidence-cited 150-item workbook in hours instead of days. Humans then triaged the results, and the open questions about AWS service semantics became an itemized agenda for an AWS Solutions Architect deep dive.

Key takeaways

  1. Guidance published faster than tooling can absorb it → treat the guidance as data. The Agentic AI Lens had 150 best practices (risk levels: 90 High, 59 Medium, 1 Low) but no console support. Smarsh extracted it into lens_data.json (ID, pillar, question, title, risk, desired outcome), cross-validated the risk levels against the lens appendix to catch extraction errors, and told the skill to never invent or renumber items. This is concepts/policy-as-data applied to an architecture-review checklist (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  2. Reasoning and rendering are split by a single JSON contract. The model emits one object per best-practice ID with status, evidence, gap, and remediation. build_report.py turns that into the workbook. The renderer validates its input: unknown best-practice IDs or invalid statuses abort the build, so "hallucinated items cannot silently ship." Row counts are checked and blank statuses flagged. Summary counts are live COUNTIF formulas, not numbers the model typed. This is a clean instance of patterns/deterministic-tool-vs-llm-judgment and concepts/structured-output-reliability (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  3. Honesty rules are part of the skill, not left to the model. Five statuses: Compliant, Partially Compliant, Non-Compliant, Not Applicable, Not Reviewed. Every definitive status must cite repository evidence. Items with no signal are marked Not Reviewed rather than guessed. The same fact must produce a consistent status across pillars. These are structural defenses against concepts/llm-hallucination (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  4. Progressive disclosure keeps the idle cost near zero. Claude finds the skill through its frontmatter description and loads the instructions, dataset, and scripts only when a review is requested (concepts/progressive-capability-disclosure). The evidence sweep is read-only over the target repo: README and architecture docs, decision records, IaC (CDK), CI/CD (GitHub Actions), and test/evaluation suites (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  5. Evaluate the evaluator before trusting it. Two synthetic fixture repos: a moderately mature LangGraph-style workload and a deliberately rough proof of concept with a hardcoded credential and a runaway-cost incident in its history. Each run was graded against 8 verifiable expectations per fixture (all 150 items present, no blank statuses, evidence for every definitive status, honest calibration, fixture-specific findings such as the hardcoded credential). With the skill: 16/16. Baseline without the skill: 8/16, with ad-hoc 18–39-item checklists that could not be traced to the lens. The eval set ships inside the skill for regression testing as the skill changes (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  6. What the real review found (an agentic-workload security checklist in practice). Strengths: per-agent IAM execution roles scoped to named runtime ARNs (concepts/least-privileged-access), SigV4-signed MCP calls through AgentCore Gateway, two-stage input validation (JSON Schema + Pydantic), a 5 MB payload cap, typed prompts and typed error contracts, exported decision audit trails, and continuous quality monitoring through AgentCore online evaluations. High-risk gaps: the gateway ingress authorizer ran with authentication disabled under a documented risk acceptance, no prompt-injection screening on message content that bad actors can influence, rate limiting left as a TODO, and no automated cost controls (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  7. Humans own judgment; the workbook is the triage surface. Engineering accepted or challenged statuses directly in the workbook. Several items collapsed into one anomaly-detection epic. One finding changed when the team showed evidence the repo sweep had missed: dependency scanning already ran on every pull request. The repository is necessary evidence, not complete evidence (concepts/human-in-the-loop) (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  8. Some gaps no repository can answer — they are service-semantics questions. Triage produced a focused AWS agenda, each item mapped to the checklist item it would close: durability and consistency of the AgentCore runtime as a session checkpoint store; retry and idempotency semantics of InvokeAgentRuntime (concepts/idempotent-operations); break-glass patterns for halting agent invocations; Bedrock Guardrails patterns for screening untrusted content; pen-test rules of engagement for AgentCore; and GuardDuty / CloudTrail coverage for agentic-runtime threat detection (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).
  9. Direction: from a one-time gate to a continuous check. Next steps: publish the skill in Smarsh's internal AI-SDLC plugin marketplace (already run on a second, architecturally different agentic repo), run well-architected checks continuously during development, make the workbook required evidence at the production milestone gate, and extend the lens-as-data approach to other Well-Architected lenses (Source: sources/2026-10-28-aws-automating-agentic-ai-well-architected-reviews-with-the-aws-c9676e8e).

Architecture

Published lens docs ──extract + cross-validate──▶ lens_data.json (150 BPs, single source of truth)
                                                        │
  "review this repo" ─▶ Claude loads skill (frontmatter description → on-demand load)
                              │
                              ├─ read-only evidence sweep of target repo
                              │    README / arch docs / ADRs / CDK / GitHub Actions / tests
                              ├─ assess 150 items with assessment-guide.md
                              │    (5-value rubric, cite-evidence, Not Reviewed > guess, cross-pillar consistency)
                              ▼
                     assessment JSON  { "AGENTSEC08-BP01": {status, evidence, gap, remediation}, ... }
                              │
                              ▼
                     build_report.py (deterministic, openpyxl)
                       validate IDs + statuses → abort on unknown; row-count check; blank-status flag
                       → workbook: dropdowns, conditional formatting, COUNTIF summary, high-risk tracker
                              │
                              ▼
          executive summary of top high-risk gaps → human triage → remediation backlog
                                                  → AWS SA deep-dive agenda (service-semantics questions)

Systems / concepts / patterns extracted

Operational numbers

  • Lens: 150 best practices / 41 questions / 6 pillars; risk levels 90 High / 59 Medium / 1 Low.
  • Status rubric: 5 values.
  • Skill eval: 2 fixtures × 8 expectations; 16/16 with skill vs 8/16 baseline; baseline checklists were 18–39 items with no lens traceability.
  • Review time: "hours instead of a multi-day manual effort" (qualitative).
  • Reviewed workload input limits: 5 MB payload cap, two-stage schema validation.
  • Smarsh scale context: 7,000+ customer organizations, 100+ communication channels, 20+ years.

Caveats

  • Customer case study co-written with the AWS account SA; includes a "Why AWS" section. The outcome figures are described as "first internal uses and the outcomes the team is tracking toward," not measured long-term results.
  • The 16/16 vs 8/16 eval is on two synthetic fixtures with expectations the team wrote. It shows the skill enforces structure and coverage. It does not show that the statuses are correct on real workloads. The real-workload correction (dependency scanning missed) shows a repository-only sweep has blind spots.
  • Some of the architecture-relevant content is a list of open questions (checkpoint-store durability, InvokeAgentRuntime idempotency, break-glass halting). The post does not give the answers.
  • The skill's prompt, rubric details, and code are not published. "To get started" links in the original were not captured in the raw scrape.

Source

Last updated · 854 distilled / 2,448 read