Skip to content

PATTERN Cited by 1 source

Assert factory object count

Encode the expected output size of a test fixture as an assertion inside the test suite itself, so future refactors that accidentally broaden the factory's object graph will fail a dedicated, obvious test instead of silently slowing every downstream test that happens to use the factory.

Problem

Test fixture libraries (e.g. FactoryBot in Rails, factory_boy in Python, testfixtures in Go) make declaring an association a one-line affair. Authors add associations incrementally; over time the transitive object graph behind a create(:x) call grows far beyond what any individual teammate realises. PlanetScale found factories creating up to 8× as many objects as expected (concepts/factorybot-object-explosion).

The cost of this growth is diffuse:

  • The engineer who adds the extra association sees no test- speed penalty in their change.
  • The cost lands on every other test that transitively uses the factory.
  • Code review doesn't catch it because the PR diff shows a one-line addition, not the transitive graph.
  • CI wall-clock slowly climbs over months without an obvious culprit.

Pattern

For each factory whose object-count is a correctness property of its fixture design, write a test that calls the factory once and asserts the expected row count.

Canonical worked example from PlanetScale (How our Rails test suite runs in 1 minute on Buildkite, 2022-01-18, Mike Coutermarsh), verbatim:

test "factory doesn't create tons of databases" do
  create(:database)
  assert_equal 1, Database.count
end

The assertion:

  • Runs in the test suite alongside normal tests — so it runs on every PR, in every branch, with the same enforcement as any other test.
  • Is a property of the factory, not of any downstream test. Factory bloat now fails one dedicated test rather than slowing dozens of diffuse ones.
  • Is local and readable — the intent ("factory should create exactly 1 database") is obvious at the test name.
  • Is cheap — asserting count after create is milli- second work.
  • Survives refactors. If someone changes :database's associations in a way that bloats the graph, the assertion fails and the CI output points directly at the regression.

PlanetScale's canonical framing:

These tests failed at first, but we worked through the factories and eventually got them down to creating the correct number of objects.

We keep these tests in our models, protecting us from any regressions when making changes to our factories.

Structural properties

  1. Invariant as test. The pattern is a specialisation of "encode correctness properties as tests in the same suite that runs on every PR" — analogous to ArchUnit for architecture invariants, schema-compatibility checks for data contracts, and linter rules for coding style.

  2. Factory-per-test granularity. One assertion per factory whose output count matters. Don't try to write one test per every factory on the team — spend the effort on the factories whose bloat has a large blast radius (the ones used transitively by many tests).

  3. Also cover join tables / many-to-many. The assert_equal 1, User.count doesn't catch a factory that creates 1 user + 10 join-table rows. Assert every table whose row count matters.

  4. Dedicated test file / model-level test. PlanetScale keeps these tests "in our models" — each model's test file gets a factory-behaviour test; no new bag of meta-tests.

When to apply

  • Test suite wall-clock dominated by fixture setup. If flamegraphs or pry-based introspection show significant time in create(...) calls, the factory audit is likely high-yield and this pattern locks in the wins.
  • Factories evolved over years by many authors. The longer the evolution, the more likely at least one association has silently cascaded.
  • Post-remediation. Apply this pattern after auditing and fixing factories; the assertion then captures the audit's output as a permanent invariant.

When not to apply

  • Tests that intentionally create large graphs — e.g. integration tests that seed realistic datasets; the assertion would be wrong for those.
  • Factories whose graph size is allowed to grow — e.g. for composite domain objects where cascading is the point.
  • Small teams / young apps — the pattern's ROI scales with factory complexity; on a 6-month-old Rails app there's usually nothing to protect yet.

Relationship to other patterns

  • Amplifies parallelism. Parallelism amplifies per-test setup cost linearly with worker count. Reducing per-test cost via this pattern gives compound gains.
  • Complements patterns/ci-parallel-over-local-serial. Parallelism is the first lever; per-test optimisation is the second; both apply on CI.
  • Sibling of architecture-test patterns (ArchUnit, module-boundary tests, SLA-as-code tests): encode an architectural property as an automated test in the same harness as unit tests.

Seen in

Last updated · 347 distilled / 1,201 read