Skip to content

PATTERN Cited by 1 source

Build without the bytes (BwoB)

Skip downloading cached action outputs to the client unless a downstream local action actually needs them; keep just the content-addressed digest. Bazel exposes this as --remote-download-minimal. Canva's adoption turned this flag — plus a trivial retry-on-cache-eviction shim — into a 2–3× lever on Bazel-step wall-clock (Source: sources/2024-12-16-canva-faster-ci-builds).

Intent

When CI workers share a content-addressed cache (e.g. bazel-remote backed by S3), a Bazel build's cache hit normally downloads the resulting artifact to the worker. For container images, test fixtures, or large native binaries, this round-trip dominates wall-clock — the client has the bytes on disk only to upload them right back out to the next action that needs them, or to discard them if nothing downstream runs locally. BwoB breaks this pattern: the client records the digest in the local in-memory graph and only fetches the actual bytes if a local action has the artifact as an input.

Mechanism

  1. Cache hit → digest, not bytes. Bazel with --remote-download-minimal records the action's output digest from the remote cache without downloading the blob.
  2. Local-action trigger pulls the bytes. If a subsequent action on the same worker needs the artifact as an input, Bazel fetches it on demand.
  3. Test action orchestration unchanged. Tests still see their inputs because Bazel fetches them when a local test action fires.

The correctness property: same build result as without the flag, because the graph's behaviour only depends on whether actions had their inputs available when they ran — not on whether cached outputs were prefetched.

Canva's numbers

Build type Before After Speed-up
Backend ~10 min ~5 min 2×
Machine learning ~6 min <2 min 3.3×

Adopted on all Bazel steps (builds and tests) after the rollout.

The cache-eviction hazard

BwoB's only failure mode: if the remote cache evicts an artifact that the local action set defers fetching until later, the later fetch returns "not found" and the build errors. (Bazel issue #10880).

Canva's fix: retry the build on that specific failure. Evictions are rare enough that retry-on-eviction costs far less than the BwoB savings:

After learning of this cache issue, we experimented with a simple workaround of retrying when those (fairly rare) cache check failures occur. It worked really well and we rolled it out.

A real-world example of "the cheap workaround beats the thorough fix" when the failure mode is rare and detection is trivial.

Preconditions

  • A shared remote cache. BwoB is a BUY-side pattern; if your cache is local-only, there's nothing to not download.
  • Hermetic actions. Without hermeticity, cache keys aren't reliable and BwoB amplifies any keying bug into a correctness incident.
  • Rare cache eviction. If your cache's GC is aggressive, the retry shim's cost climbs.

Composability

BwoB + RBE is the full "thin client" story: RBE workers share a local CAS, so even between RBE workers the artifacts don't flow over the client network. BwoB is RBE's natural successor for teams that haven't fully migrated to RBE yet.

Seen in

Last updated · 200 distilled / 1,178 read