Skip to content

PATTERN Cited by 1 source

CI parallel over local serial

Invest engineering effort in making the full-suite path fast on CI (where parallelism and large agents are cheap), not on making the full-suite path fast locally (which almost no one runs). When working locally, engineers run the test(s) directly relevant to their change; the full-suite signal comes from CI, where ample parallelism is available.

Problem

Test-suite performance is frequently framed as "the full suite should be fast everywhere", which leads teams to pursue expensive optimisations (aggressive mocking, in-memory databases, cut corners on coverage) aimed at making a full-suite local run pleasant. But:

  • Few engineers run the full suite locally. On a modern app with hundreds of tests, the full suite takes minutes on a laptop with 4-8 cores. Most engineers don't wait; they run the tests for the file they edited or a single test at a time.
  • Local machines can't compete with CI agents. A laptop has 4-16 cores; a CI agent can be 64 cores. Parallelism economics on CI always beat local.
  • Optimising for local imposes global costs. "Make it run fast locally" often means cutting fidelity (mocking the DB, stubbing background jobs, collapsing integration paths) — which reduces the value of the suite as a correctness signal.

Pattern

Split the feedback loop explicitly:

Surface Command Purpose Optimisation target
Local bin/rails test path/to/file or bin/rspec path/to/file:42 Fast edit-test cycle while working on a specific change Per-test speed; the one-test case should be fast
CI Full suite, parallelised Catch cross-cutting regressions; gate PR merge Wall-clock of the full-suite run

Consequences of this split:

  • The local environment is tuned for single-test latency. Startup cost, gem-loading, DB fixture setup — all matter for single-test latency.
  • The CI environment is tuned for full-suite throughput via parallelism (worker-count), agent size, and (later) per-test optimisation (factory audits).

Canonical framing

PlanetScale (How our Rails test suite runs in 1 minute on Buildkite, 2022-01-18, Mike Coutermarsh), verbatim:

We never run all of our application's tests in local development. It's not a good use of time and will never be as fast as running them on CI. When working locally, we'll run the tests for the single file we modified, or just a single test at a time. Then we push the commit and get feedback for the whole test suite quickly.

Our whole test suite locally takes around 12 minutes running serially on a MacBook Pro. We haven't put much effort here because it's not something our engineers ever run.

The explicit statement "we haven't put much effort here" is the pattern codified: a principled under-investment in local full-suite speed is an engineering choice, not an oversight.

Structural implications

  1. Environment-gate parallelism settings. PlanetScale's canonical if ENV["CI"] guard lets CI run with 64 workers while local defaults to serial:
if ENV["CI"]
  parallelize(workers: 64)
end

  1. Invest in CI agent capacity. PlanetScale used 64-core Buildkite agents; the pattern needs large machines to pay off. Customer-owned-agent CI models (Buildkite, self-hosted GitHub Actions runners, self-hosted GitLab runners) give the customer control of the shape.

  2. Keep tests high-fidelity. Because fast-local isn't the goal, tests can still exercise real DB, real services (via testcontainers), real integration paths — the fidelity lost by mocking-for-local-speed is preserved.

  3. Flakiness surfaces on CI, not local. Running 64-way in parallel exposes shared-state bugs the serial local run never hits. This is a cost of the pattern (investment in de-flaking infrastructure) balanced against the benefit (tests are closer to production concurrency).

  4. Local still matters. The pattern doesn't abandon local speed — it narrows the target to single-test or single-file latency. Slow application boot, heavy rails initializers, slow fixture factories for a single test — all still hurt and still need attention.

When to apply

  • Large test suites (hundreds+ of tests, multi-minute serial wall-clock).
  • Frameworks with native parallelism (Rails + minitest, RSpec + parallel_tests gem, pytest-xdist, Go's go test -parallel, Jest's --workers).
  • Customer-owned-agent CI (Buildkite, self-hosted runners) where agent shape is under engineering control.
  • Teams that use push-to-test workflows rather than mandatory-local-full-suite cultures.

When not to apply

  • Disconnected work / bad CI. If CI is slow or flaky enough that engineers must run the full suite locally before pushing, the split breaks down. Fix CI reliability first.
  • Small apps / young projects. The split's payoff scales with test-count; on a small app, a fast local full-suite is achievable and the split isn't worth the indirection.
  • Offline / air-gapped teams. If engineers frequently work without CI access, local full-suite becomes the primary feedback loop by necessity.

Relationship to other patterns

  • Sibling of patterns/assert-factory-object-count. Both are CI-focused patterns: one gets the parallelism right, the other keeps per-test cost low.
  • Prerequisite for high-N parallelism. Without the CI-vs-local split, you can't ramp workers up to 64 without penalising local developers.
  • Complements build-avoidance patterns (Bazel remote cache, pipeline-step-consolidation) — same underlying philosophy: spend engineering effort on the aggregate CI-critical-path, not on individual developer ergonomics around workflows they don't use.

Seen in

Last updated · 347 distilled / 1,201 read