Skip to content

CONCEPT Cited by 1 source

CloudFormation Verbosity Problem

Definition

The CloudFormation verbosity problem is the operational pain that AWS CloudFormation templates — even when expressive enough to describe any AWS resource set — are too verbose and too tedious to author by hand for non-trivial workloads. This verbosity is the motivating reason teams build higher-level DSL wrappers on top of CloudFormation rather than editing YAML / JSON templates directly.

Canonical description

From the 2022-04-18 Zalando ML Platform overview (sources/2022-04-18-zalando-zalandos-machine-learning-platform):

"CloudFormation templates are highly expressive and allow developers to describe even minute details. Unfortunately, CF files can become verbose and are tedious to edit manually. We addressed this problem by creating zflow, a Python tool for building machine learning pipelines."

This is a rare, explicitly-named motivation statement on the wiki for a team building a DSL on top of CloudFormation — most teams just reach for AWS CDK and move on.

Symptoms

  • Per-resource boilerplate. A single Lambda function with proper IAM, log group, env vars, VPC config can easily be 50–100 lines of YAML.
  • No type checking or IDE autocomplete in raw JSON / YAML. Typos surface at deploy time, not authoring time.
  • No first-class variables or control flow — conditionals and loops in templates use Fn::If, Fn::ForEach, intrinsic functions that are powerful but ungainly.
  • Hard to share fragments across teams. There is no Python-style package manager for CloudFormation snippets.
  • Domain-specific patterns are absent. CloudFormation has no opinion about "ML pipeline," "data pipeline," "serverless web app" — every team reinvents these on top of raw resources.

Standard fixes

  1. AWS CDK — AWS's official imperative wrapper. Write infrastructure in TypeScript / Python / Java / Go / .NET, let CDK synthesise CloudFormation. Type checking and IDE autocomplete come for free. Ships the Construct abstraction for reusable, versioned fragments.
  2. Domain-specific DSL over CDK — a team like Zalando's ML Platform wraps CDK in a narrower tool (systems/zflow) whose primitives are ML-shaped (training_job, batch_transform_job, databricks_job), not infrastructure- shaped (Lambda, IAM, Step Functions state machine). Users of the ML DSL never touch CDK or CloudFormation directly.
  3. Terraform / Pulumi / SAM / Serverless Framework — non-AWS and AWS-partner wrappers that also hide raw CloudFormation.

Zalando's zflow is an instance of fix #2 — a Python DSL that compiles to a CloudFormation template via AWS CDK underneath.

Why the pattern matters

The verbosity problem is the canonical pretext for building an internal IaC DSL. The decision to build (rather than use CDK directly) usually rests on:

  • Whether the domain (ML pipelines, data pipelines, internal services) has enough recurring shape to justify a narrower surface.
  • Whether enough teams consume the DSL to amortise build + maintenance cost.
  • Whether there is operational signal that CDK alone is still too low-level for the target users (applied scientists in Zalando's case, not infrastructure engineers).

At Zalando's scale (hundreds of pipelines created with zflow since 2019, per the 2022 post), the DSL investment clearly cleared the bar.

Last updated · 501 distilled / 1,218 read