Skip to content

PATTERN Cited by 2 sources

Tests as executable specifications

Pattern

Treat the test suite not just as a regression net, but as the behavioral specification of the system — a corpus of executable assertions that both human reviewers and AI agents read to infer what the system is supposed to do. When a test fails, the failure itself is a teaching signal: the agent reads the assertion, the inputs, the expected output, and can often refine its change without re-prompting.

Named by the 2026-03-26 AWS Architecture Blog post:

"In agentic workflows, tests do more than catch regressions, they define acceptable behavior."

"Well-written tests also act as documentation. When a test fails, the agent can infer what behavior is expected and refine its changes accordingly."

Why it works for agents

  • Tests are formally structured. Name, arrange, act, assert — the agent can parse structure in a way it can't parse a prose docstring.
  • Failure output is diagnostic. "Expected X, got Y" is a machine-readable hint; the agent uses it to steer the next edit.
  • Scope is implicit. A test suite's boundaries tell the agent what's in-contract and what isn't.
  • Specs stay executable, not wishful. Prose docs rot; tests that rot fail CI.

Test-quality properties that make this work

  • Clear names — shouldRejectNegativeAmounts teaches more than test123.
  • One assertion per test — failure tells the agent which property broke.
  • Realistic fixtures — not so stripped-down the test teaches a wrong model of the domain.
  • Tight arrange/act/assert separation — the agent can read the inputs, action, and oracle independently.

Sibling patterns in the wiki

  • executable-specification — the formal-spec tier: a compact model of the system (~1% the size) in the production language, validated via property-based testing (systems/shardstore). This pattern (tests-as-spec) is the test-suite-not-formal-spec tier of the same idea.
  • test-case-generation-from-spec — inverse direction: generate tests from a spec.
  • concepts/specification-driven-development — the broader concept this pattern is a concrete realization of at the test-suite tier.

Pairs with

  • layered-testing-strategy — unit + contract + smoke tests, each tier a different spec granularity.
  • contract-first-design — contract tests verify a formal contract the agent can also consult directly.

Caveats

  • Bad tests teach wrong behavior. If tests lie (flaky, wrong assertions, inverted expectations), the agent learns the lie. Test quality is load-bearing under this pattern.
  • Coverage gaps are invisible teachers. Absent tests look like "no spec"; the agent may invent plausible-but-wrong behavior. Pair with concepts/specification-driven-development for the formal-spec tier.
  • No measurement in the 2026-03-26 source of how much agent behavior improves under well-written tests vs poorly-written ones.

Seen in

Last updated · 766 distilled / 2,225 read