Skip to content

PATTERN Cited by 1 source

Decoupled state commit from data commit

Intent

Stop bundling state-change commits inside data-commit boundaries. In systems where both data (rows, documents, segments) and state (schema, settings, index metadata) share a commit path, bundling forces every schema update to wait for a data commit to become durable. That coupling is only justified if state changes require data-format changes — which, for backwards-compatible schema operations (field additions, new indices, live-setting tweaks), they don't. Split the commits so state changes are durable per-request and readers never need to wait for data-commit cadence to observe them.

Shape

  old:
    data commit  ─┐
                  ├──► shared commit process
    state change ─┘     (hours-of-data-committed-together
                         +state-changes-committed-together)

  new:
    data commit  ──► data-commit path
                        (flush segments, fsync, …)

    state change ──► state-commit path
                        (merge into immutable state object,
                         write to state backend, atomic swap)

Independent commit paths mean:

  • State changes take effect per-request, not per-data-commit.
  • Data commits can continue to batch for throughput without delaying state changes.
  • Failure modes don't cross: a failed data commit doesn't lose a state change (and vice versa).

Canonical instance: Yelp Nrtsearch 1.0.0

Pre-1.0 Nrtsearch bundled state into the data commit path. Per the 1.0.0 post:

"The commit of index state and data were coupled together. This was unnecessary, since the only allowed state changes were backwards compatible with previous data. Specifically, new fields can be added to the index, but existing fields cannot be removed or modified."

"State changes were not durable until after a commit request. This meant that an update could be lost if the primary server restarted between the state modification and the commit."

Two distinct wins called out:

  1. No unnecessary bundling. Backwards-compatible state changes don't need to ride data commits.
  2. Per-request durability. State changes become durable as part of the request that issues them, not on a later data-commit boundary — so a primary restart between modification and commit no longer loses the change. "Clients will also no longer see state values applied to the primary that have not yet been committed."

Composes with:

Why the bundling was there in the first place

Bundling state+data commits is a common simplification in the early design of a system — one commit process, one durability guarantee, one crash-recovery story. The coupling only becomes painful once:

  • state changes become frequent enough that users notice the latency;
  • "I lost my schema update because the primary restarted" becomes a real operational story;
  • cluster propagation of state changes (tied to data-commit cadence) creates observable pressure on ops to batch schema updates.

At that point, splitting the commits is the right move — the simplification paid for itself early but is no longer worth the operational drag.

Preconditions

  • State changes must be backwards-compatible with existing data. (For Nrtsearch: field additions, not field removals or type changes.) Breaking changes still require a coordinated migration and can't be cleanly decoupled from data.
  • The state backend supports per-request durability (S3 PUT, EBS fsync, transactional write to a config store).
  • The state representation supports atomic swap after commit (concepts/immutable-index-state) so readers never see a partially-committed state.

Seen in

Last updated · 550 distilled / 1,221 read