Skip to content

CONCEPT Cited by 1 source

Declarative workflow specification

Definition

A declarative workflow specification is a structured document (typically JSON or YAML) that describes what a data pipeline should do — datasets, field mappings, transformation capabilities, and constraints — without containing any processing logic. It separates workflow intent from execution, enabling:

  • Pre-execution validation — the specification can be checked against a schema and a capability registry before any processing begins.
  • Versioning and traceability — specifications are explicit, immutable records of workflow intent, supporting auditing in regulated environments.
  • Separation of duties — business/domain users author intent; the system (a composer) produces execution artifacts.

Example structure

{
  "source": { "dataset": "raw_orders" },
  "target": { "dataset": "orders_clean" },
  "mappings": [
    {
      "source_field": "order_date",
      "target_field": "order_date",
      "transformation": { "capability": "format_date" }
    }
  ]
}

Each mapping links a source field to a target field and references a named capability from a concepts/capability-registry. The specification contains no executable code.

Relationship to other declarative concepts

This concept is the data-pipeline instantiation of the broader principle that declarative interfaces (specify what, not how) improve composability, governance, and toolability. Related declarative concepts in this wiki include concepts/declarative-vs-imperative-stream-api, concepts/declarative-lifecycle-api, and concepts/declarative-agent-definition.

(Source: sources/2026-07-09-aws-specification-driven-composition-for-flexible-data-workflows)

Seen in

Last updated · 570 distilled / 1,718 read