Skip to content

AWS 2026-10-26

Read original ↗

AWS: Recommended resilience testing in the next generation of AWS Resilience Hub

Summary

Recommended resilience testing is a next-generation AWS Resilience Hub capability that turns resilience-policy definition into evidence-backed validation without hand-building fault experiments. It reuses what Resilience Hub already knows about a service — resources, dependencies, and resilience policy — to auto-generate AWS FIS experiments, inject faults, and evaluate results against the service's success criteria and RTO, producing a pass/fail outcome and an auto-generated report per run. It ships four pre-configured tests at launch — Availability Zone: recovery, Dependency validation, Multi-Region: isolation, and Multi-Region: recovery — selected from the service's architecture and policy. The article's deep dive is dependency validation: classify each dependency soft (service degrades but survives → test should pass) or hard (service fails without it → test may fail, informatively), then block connectivity at the compute layer (100% packet loss) and read success alarms to confirm the classification was right. The removed barrier is expertise: "what used to be manual experiment design becomes an out-of-the-box test with built-in success evaluation based on your policy."

Key takeaways

  1. Policy-aware, auto-generated experiments. Resilience Hub creates FIS experiments from the service's discovered resources + dependencies + resilience policy, injects faults, and judges pass/fail against success criteria and the policy's RTO — no manual experiment design. Each run auto-generates a console-accessible report (Source: sources/2026-10-26-aws-recommended-resilience-testing-next-gen-resilience-hub).
  2. Four recommended tests at launch, chosen by architecture + policy:
  3. Availability Zone: recovery — injects the symptoms of a single-AZ power interruption (compute, networking, storage, database in that AZ); confirms detection, continued operation in healthy AZs, and return to healthy within RTO. Can surface unknown single-AZ dependencies.
  4. Dependency validation — blocks dependencies within the Region to see how the service behaves when they are unavailable; validates soft dependencies degrade gracefully, surfaces dependencies that are secretly hard.
  5. Multi-Region: isolation — blocks connectivity between two Regions (network traffic, cross-Region replication, selected dependencies); validates the service operates independently of the other Region. Applies to both active-active and active-passive; a precursor to Multi-Region: recovery.
  6. Multi-Region: recovery — introduces failures into dependencies in the selected Region; via success alarms, confirms recovery within RTO in the recovery Region. You perform the recovery action yourself (trigger a failover or an ARC Region switch plan).
  7. Three-level model: test template → test → test run. A test template is the pre-configured recommendation (fault actions, target resource types, parameters). A test is your configured instance (success alarms, stop conditions, settings). A test run is one execution scoped to current resources that yields a pass/fail + report. Runs are repeatable quarterly / monthly / on demand (Source: sources/2026-10-26-aws-recommended-resilience-testing-next-gen-resilience-hub).
  8. Two evaluation shapes keyed to test type. Recovery tests (AZ recovery, Multi-Region recovery) pass when all success criteria return to healthy within the policy RTO and hold until the test ends — validating recovery was fast enough, not just that it happened. Sustained tests (Dependency validation, Multi-Region isolation) pass when all success criteria hold their thresholds from fault injection through the end of the test — proving the service stayed healthy throughout the impairment.
  9. Choose success criteria that reflect customer experience, not resource health. Pick order success rate or end-to-end availability, "rather than the state of individual resources. The impaired resources are expected to be unhealthy. What you are validating is whether your service as a whole absorbed the failure." Additional observability-only alarms can be attached without affecting the outcome. This is the same stop-condition discipline FIS already teaches — alarm on customer impact, never on the effect you inject.
  10. Soft vs hard dependency classification is the core of dependency validation. Soft (analytics API, notification service, feature flags, recommendation engine): service degrades but continues → test should pass. Hard (primary database, authentication service, credential exchange): service fails without it → test might fail, which is expected and informative. The test's job is to confirm or refute the classification.
  11. Dependencies are defined two ways. Turn on dependency discovery (DNS query-log analysis, 35-day look-back + continuous polling, surfaces unexpected cross-Region calls, critical third-party deps, and rarely-used services) or manually enter DNS endpoints. The discovered soft dependency with the highest query volume is pre-selected by default; you can choose up to 10 dependencies per test.
  12. Dependency-validation faults are compute-layer packet loss via FIS, auto-matched to compute type. Three FIS actions inject 100% packet loss to the selected dependencies — aws:ssm:send-command (EC2 instances), aws:ecs:task-network-packet-loss (ECS tasks on EC2 and Fargate), aws:eks:pod-network-packet-loss (EKS pods). When the test starts it matches actions to discovered resources; actions with no matching target are skipped, so the test adapts to EC2 / ECS / EKS / mixed environments. Setup differs per compute type (SSM Agent on EC2, an SSM Agent sidecar in the ECS task definition, a Kubernetes service account for EKS pods).
  13. Worked contrast — same service, same alarms, two classifications. Blocking soft deps (Payment Gateway, Notification Service) → both success alarms (canary-heartbeat-..., canary-order-flow-...) stayed green the whole run → PASSED; graceful degradation confirmed. Blocking hard deps (Amazon Cognito + AWS STS) → heartbeat stayed green but order-flow went red immediately and stayed red → FAILED — and the failure was contained (no cascade; the service kept responding while only the auth-dependent business flow failed). A failure that proves containment is a successful experiment.
  14. Results are a 2×2 decision framework, not a verdict. Soft→pass = classification correct, no action. Soft→fail = misclassified-as-soft or a degradation-path gap → reclassify or fix. Hard→fail = classification confirmed → verify the failure is graceful (no cascade/corruption/hung connections), consider adding retry/fallback to make it soft. Hard→pass = possibly misclassified as hard, or the dependency isn't actually being exercised → investigate.

Systems / concepts / patterns extracted

  • Primary subject system: AWS Resilience Hub — recommended resilience testing is a next-gen capability of it (policy-aware test generation + built-in pass/fail evaluation + reports).
  • Execution substrate: AWS FIS — all four tests are built on FIS; dependency validation introduces three new FIS packet-loss actions.
  • Compute targets: Amazon ECS (+ Fargate), Amazon EKS, Amazon EC2 — the per-compute fault actions.
  • Supporting: CloudWatch alarms (success criteria + stop conditions), Amazon S3 (report + log destination), IAM roles (execution / orchestrator+target for multi-account), Route 53 Resolver (DNS-log dependency discovery), ARC (operator-run Region switch in Multi-Region recovery), Amazon Cognito + AWS STS (the hard-dependency example). AWS STS and Payment/Notification demo services named plain-text (no page).
  • Concepts: concepts/chaos-engineering (the discipline, now packaged as out-of-the-box policy-evaluated tests), concepts/graceful-degradation (the property soft-dependency validation proves), RTO (the pass bar for recovery tests), concepts/regional-failover (active-active / active-passive multi-Region tests), concepts/blast-radius (dependency selection + contained failure), concepts/cascading-failure (what a contained hard-dep failure avoids), concepts/static-stability (operate in healthy AZs through impairment).
  • Patterns: patterns/staged-rollout — "start with a single soft dependency and expand from there" is the escalate-severity-in-phases discipline applied to dependency blocking.
  • Recorded as prose/tags, NOT minted (single source, maps into existing vocabulary or an implementation detail): soft-vs-hard-dependency-classification, dependency-validation-test, recovery-test-vs-sustained-test-evaluation, success-alarm-on-customer-experience-not-resource-health, classification-as-hypothesis-decision-framework, compute-layer-packet-loss-auto-matched-to-target, skip-action-with-no-matching-target. These map into concepts/chaos-engineering + concepts/graceful-degradation and the FIS experiment model. (Lint can promote soft/hard dependency classification once it earns ≥3 citations — currently the closest existing anchor is aws-resilience-hub's hard/soft update-dependency --criticality lever from the 2026-10-13 source.)

Operational numbers

Figure Value
Recommended tests at launch 4 (AZ recovery, dependency validation, Multi-Region isolation, Multi-Region recovery)
Dependency-discovery look-back window 35 days (DNS query-log analysis) + continuous polling
Max dependencies blocked per test 10
Success criteria + additional alarms limit 5 per configuration update
Stop-condition alarms up to 5 (same account + Region as fault injection)
Test duration 1 minute – 12 hours (default 30 minutes) + a few minutes to evaluate
FIS fault 100% packet loss at the compute layer to selected dependencies
Dependency-validation FIS actions aws:ssm:send-command (EC2), aws:ecs:task-network-packet-loss (ECS/Fargate), aws:eks:pod-network-packet-loss (EKS)
Pre-selected dependency discovered soft dependency with the highest query volume
Availability all AWS commercial Regions where next-gen Resilience Hub is supported

Caveats

  • Feature-walkthrough / guidance post, not a production retrospective. The two "test runs" (soft PASSED, hard FAILED) are illustrative examples with demo services (Payment Gateway, Notification Service) and demo canaries, not measured customer incidents. Ingested for the reusable resilience-testing vocabulary (recovery-vs-sustained evaluation, soft/hard dependency classification, policy-RTO pass bar, auto-matched compute-layer fault actions).
  • Classification is the hypothesis, not ground truth. A passing test can still surface a misclassified dependency and a failing test can confirm a genuinely-critical one — results must be read against how you classified, per the 2×2 framework.
  • Dependency discovery is history-bounded (35-day window, continuous polling) and inherits DNS-discovery blind spots noted on the Resilience Hub page (direct-IP, Lambda-without-VPC, infrequent calls); manual DNS entry is the fallback.
  • Per-compute setup required before packet-loss actions work (SSM Agent on EC2, SSM Agent sidecar in the ECS task definition, a Kubernetes service account for EKS pods); unmatched actions are silently skipped.
  • Multi-Region: recovery does not perform the failover for you — you run the recovery action (failover or ARC Region switch plan); the test only validates recovery-within-RTO through success alarms.
  • Updating test settings can change required IAM permissions (vary by trust policy, fault actions, and options) — always re-review the role.

Source

Last updated · 854 distilled / 2,448 read