PATTERN Cited by 1 source

Dynamic pipeline assembly

Definition

Dynamic pipeline assembly is the runtime pattern where a composer reads a declarative specification, validates it against a capability registry, and constructs an executable workflow (e.g., a Step Functions state machine) on the fly — rather than using a statically defined pipeline. The assembled pipeline is a system artifact (execution code) derived from a business artifact (the specification).

How it works

1. Specification arrives (e.g., uploaded to S3, triggered by EventBridge schedule, or API call).
2. Composer Lambda parses and schema-validates the specification.
3. For each referenced capability, the composer queries the registry to verify existence and retrieve metadata (ARN, version, input/output schema).
4. Composer generates an Amazon States Language (ASL) definition mapping each capability to a sequential step.
5. Composer creates/starts a Step Functions state machine from the ASL.
6. Step Functions orchestrates Lambda capability processors in sequence.

Trade-offs

• Pro: New pipelines can be created without code changes — just a new specification.
• Pro: The composer can apply security/compliance checks at assembly time before any data processing occurs.
• Con: Runtime assembly introduces latency at pipeline start (registry lookups, ASL generation).
• Con: Debugging a dynamically generated state machine is harder than a static one.

(Source: sources/2026-07-09-aws-specification-driven-composition-for-flexible-data-workflows)

Seen in

• sources/2026-07-09-aws-specification-driven-composition-for-flexible-data-workflows — AWS Architecture Blog (2026-07-09): composer Lambda dynamically generates Step Functions state machines from validated specifications.