Skip to content

CONCEPT Cited by 1 source

Hermetic build

Definition

A hermetic build (or test) declares every input it depends on — source files, tool binaries, environment variables, OS libraries — and runs in a sandbox that blocks access to anything undeclared. Same inputs → same output, reproducibly, across developer machines, CI workers, and remote executors.

Hermeticity is the precondition for three independent wins:

  • Caching. The input set fully defines the output, so the output can be cached by input-hash (concepts/content-addressed-caching).
  • Parallelism. No hidden shared state means actions can run in parallel on one host or fan out to remote workers.
  • Reproducibility. A green build on CI is green on a laptop.

Why non-hermetic CI steps compound at scale

From Canva's retrospective, allowing arbitrary commands/scripts as CI steps was author-friendly but scaled badly:

Steps weren't hermetic or deterministic, wrapped as a Bazel build or test. So they can't run in parallel (affect each other's output), can't be cached (no full definition of inputs), can't be easily isolated, can cause flakes by leaking state into other steps, and are hard to run locally.

The consequence: each step took a whole EC2 instance to avoid cross-step interference, which pushed Canva to 3000 jobs/build × >1000 builds/day and bottomed out their parallelism gains.

Hermetic tests: the TestContainers lever

Shared-localstack integration-test harnesses are the canonical non-hermetic pattern: all tests run against one set of storage containers, so ordering and resource contention leak across tests. Flakes follow.

The Canva fix: each test gets its own TestContainers-orchestrated container set, inside a Bazel sandbox. Each test declares its storage dependencies. Result — tests become cacheable, parallel-safe, and far less flaky. (See systems/testcontainers.)

Extended up the stack: per-service TestContainer implementations + per-service launch-validation tests → hermetic E2E environments composed from those service containers. Deployment failures shift left to CI.

Hermeticity tax

Declaring every input costs up-front effort:

  • Every source file listed.
  • Every tool binary pinned.
  • Sandbox symlinks per input — expensive on node_modules (thousands of files per action).
  • Bazel startup dominated by loading/analysing the graph (Canva: 900K nodes → minutes).

From the Canva post:

[Bazel] is hard to migrate builds and tests into because you must declare every input that a rule depends on, which requires a significant amount of time, effort, and some Bazel-specific knowledge.

And: RBE compatibility requires the worker pool to produce the same inputs as local execution — non-hermeticity surfaces as RBE bugs.

Seen in

  • sources/2024-12-16-canva-faster-ci-builds — non-hermetic steps identified as a primary CI-inefficiency driver; TestContainers rollout for backend integration + E2E tests made them cacheable and reliable.
Last updated · 200 distilled / 1,178 read