Skip to content

PATTERN Cited by 1 source

Specification-driven composition

Definition

Specification-driven composition is a data-pipeline architecture pattern that separates workflow intent from processing logic. Instead of embedding transformation rules directly in scripts or application code, the system:

  1. Describes pipeline behavior in a structured specification (JSON/YAML) — declarative, versioned, auditable.
  2. Uses a composer to validate the specification against a capability registry and assemble a runnable pipeline.
  3. Executes the assembled pipeline as a sequence of reusable transformation processors.

The specification expresses what the workflow should do in domain terms; the composer translates intent into an execution artifact (e.g., a Step Functions state machine); processors perform the actual transformations.

Problem it solves

Script-based data pipelines accumulate hidden costs at scale:

  • Duplication — transformation logic is copied across pipelines; small changes cascade.
  • Governance gaps — workflow intent is buried in code; auditing requires reading scripts.
  • Slow onboarding — adding a new dataset requires modifying and redeploying code.
  • Late validation — errors surface only during processing, not before invocation.
  • Separation-of-duties violations — in regulated environments, business users shouldn't need to modify execution code to express intent.

Core components

Component Responsibility
Specification Declares datasets, field mappings, transformation capabilities. No processing logic.
Composer Validates spec → looks up capabilities in registry → assembles pipeline. Does not transform.
Capability registry Governed metadata store of reusable transformation functions (IDs, schemas, ARNs, versions, permissions).
Capability pipeline Sequence of processors that each perform one transformation step.

When to apply

  • ≥3–5 data transformation workflows with overlapping logic.
  • Regulated environments requiring traceability, auditability, and separation of duties (GxP, financial reporting, clinical trials).
  • Multi-source data integration where transformation capabilities repeat across variants.
  • Teams wanting to accelerate dataset onboarding (weeks → days).

Trade-offs

Pro Con
Clear governance and auditability Adds architectural complexity
Reuse reduces duplication Requires disciplined registry governance
Faster onboarding once library matures Upfront investment in specification format design
Strict separation of intent vs execution Over-engineered for <3 simple pipelines
AI-assistable capability discovery Specification format is another thing to version and maintain

Reference implementation (AWS)

S3 (spec upload) → Lambda (composer) → OpenSearch (capability registry)
                                      → Step Functions (assembled pipeline)
                                      → Lambda processors (capabilities)
                                      → CloudWatch (traces)

(Source: sources/2026-07-09-aws-specification-driven-composition-for-flexible-data-workflows)

Relationship to other patterns

  • Extends concepts/separation-of-concerns to data pipeline design across three explicit layers (intent / composition / processing).
  • The composer is analogous to a compiler: specification is the source, the state machine is the target artifact.
  • Leverages concepts/event-driven-architecture for trigger flexibility (S3 events, schedules, API calls).
  • The capability registry concept (concepts/capability-registry) is reusable beyond this pattern — any system where pluggable processors need governance.

Seen in

Last updated · 570 distilled / 1,718 read