PATTERN Cited by 1 source
Three-layer pipeline separation¶
Definition¶
Three-layer pipeline separation organizes a data transformation workflow into three distinct layers with clear responsibilities:
| Layer | Responsibility | Artefact |
|---|---|---|
| Intent | Defines what the workflow does via a declarative specification | JSON/YAML specification |
| Composition | Validates specs, resolves capabilities, assembles pipeline | Composer (e.g., Lambda) + registry |
| Processing | Executes the assembled pipeline of transformation steps | Capability processors |
Each layer evolves independently: specifications can change without modifying processing code; new capabilities can be registered without touching the composer; the composer's assembly logic can be improved without affecting either intent or processing.
Rationale¶
The pattern is the data-pipeline instantiation of concepts/separation-of-concerns:
- Business users own the intent layer (what should happen).
- Platform engineers own the composition layer (how it's assembled).
- Capability developers own the processing layer (individual transformations).
This maps cleanly to regulated-environment role boundaries where separation of duties is required by policy (e.g., GxP).
(Source: sources/2026-07-09-aws-specification-driven-composition-for-flexible-data-workflows)
Seen in¶
- sources/2026-07-09-aws-specification-driven-composition-for-flexible-data-workflows — AWS Architecture Blog (2026-07-09): the core structural pattern underlying specification-driven composition.