Skip to content

CONCEPT Cited by 1 source

Test feedback loop

The wall-clock time from push-commit to full-suite pass/fail signal — a DevEx-first-order primitive that directly shapes how engineers develop, rebase, and ship changes. Slow test feedback loops distort engineering behaviour: engineers context-switch while waiting, avoid running the suite, or skip writing tests at all.

Definition

The test feedback loop is the round trip:

  1. Engineer pushes a commit.
  2. CI pulls it, runs the full test suite.
  3. Engineer sees pass / fail signal.

The primary metric is wall-clock duration of step 2. Fast loops enable tight iteration; slow loops forcibly batch work.

Why it's a first-order DevEx primitive

Team productivity compounds on loop speed. PlanetScale's canonical datum (How our Rails test suite runs in 1 minute on Buildkite, 2022-01-18, Mike Coutermarsh):

Everyone on our team has worked in the past on Rails applications with slow test suites and knew how much it hurts team productivity. As our app has grown, we have continually invested time into keeping our test suite fast. We know how much a quick feedback cycle pays off for our team and a little extra work on it makes every feature we build easier.

The investment rationale is compounding: every feature, every refactor, every bug fix pays the test-suite tax. Dropping the loop from 12 min to 1 min gives every engineer back ~11 minutes per iteration — and engineers iterate dozens of times per day.

Where to spend the time — CI, not local

PlanetScale's canonical rule ("we never run all of our application's tests in local development. It's not a good use of time and will never be as fast as running them on CI") splits the feedback loop into two surfaces:

  • Local: engineer runs single test or single file while editing (fast, narrow, high-signal for the change).
  • CI: engineer runs full suite after push (broad, catches cross-cutting regressions, runs in parallel on a large machine the engineer doesn't have locally).

This is the CI- parallel-over-local-serial investment rule. Optimising the full-local-suite path is wasted effort because no one runs it.

Levers that shorten the loop

In decreasing order of typical impact:

  1. Parallelism (concepts/test-parallelism-worker-count) — spread work across more cores. PlanetScale: 12 min → 3-4 min by going from 2 to 64 workers.
  2. Per-test setup optimisation — remove factory-explosion cost per test. PlanetScale: 3-4 min → ~1 min by auditing factories.
  3. Test selection — only run tests affected by the change (not exercised in this PlanetScale post; Canva's sources/2024-12-16-canva-faster-ci-builds canonicalises this with input-hash-based selection).
  4. Hardware — faster CPUs, faster disks, faster networks to dependencies.

Worked numbers — PlanetScale 2022

  • Local serial: ~12 min (on MacBook Pro; not actually optimised).
  • CI, 64 parallel workers, pre-audit: 3-4 min.
  • CI, 64 parallel workers, post-factory-audit: ~1 min.

The ~11-minute improvement per push, times N engineers × M pushes/day, is the aggregate productivity gain. The post doesn't quantify the dollar-value but the qualitative claim is strong: "a little extra work on it makes every feature we build easier."

Seen in

Last updated · 347 distilled / 1,201 read