Skip to content

AWS

Specification-driven composition for flexible data workflows

Summary

This AWS Architecture Blog post introduces specification-driven composition, a pattern for building flexible data transformation workflows that separate workflow intent from processing logic. Instead of embedding transformation rules in scripts (which leads to duplication, governance gaps, and slow onboarding of new datasets), the system uses structured JSON/YAML specifications that describe what a pipeline should do, a composer that validates and assembles pipelines from a capability registry, and reusable transformation processors. The reference implementation uses S3 event notifications → Lambda composer → OpenSearch capability registry → Step Functions state machine → Lambda capability processors.

Key takeaways

  1. Script-based pipelines don't scale — as pipelines grow, transformation logic is duplicated across scripts, making changes cascade and governance difficult, especially in regulated environments (healthcare, finance, life sciences).

  2. Three-layer architecture — the pattern separates concerns into: (a) an intent layer (the specification defining workflow behavior), (b) a composition layer (validates specs and assembles pipelines), and (c) a processing layer (runs the assembled sequence of transformations).

  3. Specifications are declarative, versioned, and auditable — they describe datasets, field mappings, and transformation capabilities in structured JSON/YAML without containing processing logic; this provides a clear, traceable record of workflow intent that supports regulated-environment governance.

  4. Capability registry as a governed artifact — transformation functions live in a metadata registry (here, OpenSearch) with identifiers, input/output formats, versioning, and permission boundaries. OpenSearch enables full-text and semantic search for capability discovery by intent rather than exact identifier.

  5. Composer compiles intent into execution code — the composer reads a specification, validates referenced capabilities exist, retrieves their metadata (including Lambda ARNs), and generates an AWS Step Functions ASL state machine. Critically, the composer does not perform transformations — it only assembles.

  6. Separation of duties for regulated environments — in GxP and similar regulated contexts, business users author specifications (intent) while the composer generates execution code, maintaining strict separation between who defines what and who/what produces how.

  7. Multiple invocation triggers — the same composed pipeline can be triggered by S3 events, EventBridge schedules, API calls, direct Step Functions StartExecution, or upstream pipeline output, enabling reuse without re-authoring or re-approving the workflow.

  8. Data sensitivity propagation — specifications can tag sensitive fields (e.g., PHI), and the registry declares each capability's sensitivity behavior (preserves, clears, or sets classification). The composer validates these combine correctly and generates masking artifacts (e.g., Lake Formation column grants) automatically.

  9. Not for simple pipelines — the pattern adds complexity overhead; it is most valuable when managing ≥3–5 workflows, and particularly suited to regulated reporting, multi-source data integration, and reusable ETL frameworks.

  10. Onboarding acceleration — teams adopting this pattern report onboarding new datasets in days rather than weeks because most required capabilities already exist in the registry.

Architecture

┌─────────────────────────────────────────────────────┐
│  INTENT LAYER       Specification (JSON/YAML)       │
│                     - source/target datasets        │
│                     - field mappings                 │
│                     - capability references          │
└─────────────────────────────┬───────────────────────┘
                              │ S3 upload event
┌─────────────────────────────▼───────────────────────┐
│  COMPOSITION LAYER  Lambda Composer                  │
│                     ├── validates spec (schema)      │
│                     ├── queries OpenSearch registry  │
│                     ├── resolves capability ARNs     │
│                     └── generates ASL state machine  │
└─────────────────────────────┬───────────────────────┘
                              │ Step Functions workflow
┌─────────────────────────────▼───────────────────────┐
│  PROCESSING LAYER   Lambda Capability Processors    │
│                     (format_date, normalize_currency,│
│                      validate, enrich, …)           │
│                     → CloudWatch traces             │
└─────────────────────────────────────────────────────┘

Operational numbers

  • Pattern is cost-effective for ≥3–5 workflows; below that, script-based pipelines are simpler.
  • Dataset onboarding time reduction: weeks → days once capability library is established.
  • Security: SSE-KMS on S3, TLS on all transport, node-to-node encryption on OpenSearch, IAM per-source-bucket access control.

When to use

  • Regulated data pipelines (clinical trials, financial reporting, GxP compliance).
  • Multi-source data integration where transformations repeat across variants.
  • ETL frameworks where governance, traceability, and auditability are required.
  • Scenarios needing strict separation of duties (business users author intent, system produces execution artifacts).

Caveats

  • Introduces architectural complexity not justified for fewer than 3–5 pipelines.
  • Requires disciplined capability registry governance (version control, CI/CD validation, approval workflows for new capabilities).
  • The specification format and validation logic must be designed and maintained as a first-class engineering artifact.
  • Pre-existing script-based pipelines need refactoring investment to extract reusable capabilities.

Source

Last updated · 570 distilled / 1,718 read