Skip to content

CONCEPT Cited by 1 source

Mock object maintenance cost

Definition

Mock object maintenance cost is the load-bearing but usually- unaccounted engineering cost associated with keeping test doubles (mocks, stubs, in-memory fakes) consistent with the behaviour of the real systems they replace. The cost compounds along three axes:

  1. Volume of test-infrastructure code. Across evaluated teams, Thoughtworks reports that "mock objects account for 20-30% of test code. That's not test coverage — it's test infrastructure." (Source: sources/2026-04-30-databricks-backstage-with-lakebase)
  2. Divergence from production behaviour over time. The behaviour of the real system changes (bug fixes, schema evolution, query-planner upgrades, new constraints); the mocks stay where they were. This gap is invisible until tests stop catching real bugs.
  3. False confidence. Passing tests that run against mocks produce a reassurance signal that is partially decorrelated with production correctness. Teams ship confident, then get surprised during deployment / staging smoke tests / production rollout.

Canonical framing (Thoughtworks, 2026-04-30)

"In our experience across multiple partner teams evaluating this workflow, mock objects account for 20-30% of test code. That's not test coverage — it's test infrastructure. Infrastructure that diverges from production behavior over time, creating false confidence. When branching a production-equivalent database costs nothing, mocking becomes the expensive choice."

The final sentence is the load-bearing claim: mock-as-cheap- substitute is only cheap when the alternative (real database in tests) is expensive. When the alternative becomes effectively free (instant database branching on a copy-on-write substrate), the cost-benefit flips.

What the cost actually looks like

Imagine a team with:

  • ~200 repository interfaces (UserRepository, OrderRepository, ServiceCatalogRepository, ...).
  • ~200 corresponding mock implementations (MockUserRepository, MockOrderRepository, ...).
  • ~1,000 test files using these mocks.

Every schema change, query addition, query-shape change, or repository-method addition requires editing a subset of the mock implementations. Each edit is usually mechanical but non-localised — the mock must stay consistent with the real implementation it represents. When someone misses an update, tests pass against stale mocks while production fails.

Divergence failure modes

  • Schema evolution. Real DB adds a NOT NULL column; mock implementation returns a struct with a default-zeroed field; test passes, production crashes on insert.
  • Query-planner behaviour change. Real DB's new planner version picks a different index, changing query latency profile or lock ordering; mock has no latency or locks at all.
  • Constraint additions. Real DB has a new unique constraint; mock has no constraint at all; test inserts duplicates without error, production rejects them.
  • Transaction semantics. Real DB has serializable isolation + explicit locks; mock has no transactions at all.
  • Performance characteristics. Mock is O(1) dict-lookup; real DB is O(log n) B-tree lookup; tests don't catch performance regressions because mocks don't have representative performance.

Each of these produces a test passes but production fails pattern — the highest-cost class of bug because it bypasses the test suite's trust signal.

Why mocks still dominated pre-branching

The structural argument for mocks was cost:

  • Booting a test database per test-run was slow (seconds to minutes).
  • Booting a test database per test was infeasible (minutes per test; 1,000 tests = unrunnable suite).
  • Shared test databases had flaky test isolation (one test's inserts visible to another).
  • In-memory substitutes (H2, SQLite) were cheaper-but- different (the same divergence problem in a different coat).

So mocks were the pragmatic choice — cheap enough to run per test, consistent enough with the real API to pass most behaviour checks. The consequence is the three-axis cost canonicalised here.

What changes with cheap branching

When database branching is sub-second + free, each of the pre-branching arguments collapses:

  • Per-test database: a test creates a branch, runs against it, discards. On Lakebase the 63 MB Backstage catalog branches in 1.09 s — tolerable per-test-class, not per-individual-test yet.
  • Per-PR database: a CI job spins up a branch for the full test suite; fresh state every run.
  • Per-QA-tester database: every QA engineer has their own branch they can corrupt / reset at will.

When the alternative is cheap, the 20-30% of test- infrastructure code becomes deletion candidates, not a cost of doing business. See patterns/database-branch-per-test-over-mocking.

What mocks are still useful for

Mocks aren't universally obsolete under cheap branching:

  • External-service mocks. Payment gateways, email services, third-party APIs — these aren't branchable via a database primitive; mocks (or contract tests) remain the answer.
  • Failure-injection. A mock can deterministically raise a specific exception; a real DB can't be told "fail with constraint violation" on demand as precisely.
  • Unit-level-precision on pure logic. A pure function that takes a User struct should still be tested by passing in constructed structs, not by querying a DB.

The argument is narrow and specific: mock objects that stand in for the database are the 20-30% worth of infrastructure branching deprecates.

Seen in

  • sources/2026-04-30-databricks-backstage-with-lakebase — canonical first wiki instance. Thoughtworks articulates the 20-30% test-infrastructure cost + the divergence / false-confidence failure modes, then frames the collapse of the cost-benefit trade-off when copy-on-write database branching makes real-DB test environments effectively free. Paired with the 1.09-second / 3.78-second branching
  • PITR numbers to make the cost-comparison concrete.
Last updated · 439 distilled / 1,268 read