Skip to content

CONCEPT Cited by 1 source

Eventual consistency too slow for allocation

Eventual consistency too slow for allocation is the failure mode where an eventually-consistent store's convergence time exceeds the rate at which an exclusive resource is claimed, and the store's reads return "available" values that have already been claimed by a different requester within the consistency window. The result is race conditions + resource overallocation — two callers who both saw the resource as free both proceed to claim it, and either both succeed (resource is double-allocated) or one's claim gets overwritten (the resource is held by somebody but the bookkeeping says somebody else).

The arithmetic

Let:

  • R = resource claim rate (claims/sec)
  • T = convergence-time floor of the store (e.g. cache TTL)
  • N = number of in-flight resources (e.g. running containers)

If R × T > 1 claim per resource lifetime, the eventually- consistent store's read will be stale by definition during the convergence window. The fraction of stale reads is bounded below by R × T / N. As R grows (demand spike), or as T fails to shrink (store's design floor), the failure mode appears.

The flip from works to doesn't work is one-sided: it manifests under demand spikes (when scaling matters most) and disappears under steady-state low load.

Cloudflare KV canonical instance

Cloudflare's 2026-05-13 Browser Run migration post canonicalises this failure mode at production altitude (Source: sources/2026-05-13-cloudflare-browser-run-now-running-on-cloudflare-containers-its-faster). Browser Run tracked container allocation state in Workers KV. As demand from AI agent builders ramped up:

"AI agent builders discovered Browser Run and quickly brought request volumes outpacing our existing capacity. We quickly hit the limits of how quickly we could adjust our pool capacity to serve this new demand with a scalable approach. KV's eventual consistency of around 30 seconds was becoming a bottleneck on our critical request path. You might check KV, see a container as 'available,' but by the time you route to it (30 seconds later), it's already claimed. That lag creates race conditions and overallocation of browsers, severely limiting how fast we could scale to meet demand spikes."

The post explicitly notes that the recently reduced KV minimum cache TTL (60s → 30s) "is still too high" — a meaningful platform improvement does not necessarily close the gap for an allocation hot-path workload, because the floor on T is set by the store's design (edge-cache replication strategy), not by configuration that can be tuned arbitrarily down.

Distinction from sibling concepts

  • concepts/eventual-consistency — the parent property. Most workloads tolerate it (caches, read-mostly serving, background analytics). This concept is the subset of workloads that don't.
  • concepts/check-then-act-race — TOCTOU at single-process altitude. The eventually-consistent-too-slow failure is the distributed version: the check and the act are at the same process, but the check reads an eventually-consistent view that lags reality.
  • concepts/cache-ttl-staleness-dilemma — the related staleness-vs-coherence tension on cache TTL. The allocation- too-slow concept is the one-direction form: only the staleness side bites; you cannot tune the cache TTL low enough to match R because T has a design floor.

Symptoms — diagnostic

  • Workload is allocation-shaped (claim a unique resource for exclusive use of one caller; release on completion).
  • Read-after-write coherence is required across requesters (one caller's claim must be visible to other callers' next reads).
  • Demand bursts faster than the store's convergence window.
  • Stale-read consequence is bad (overallocation, double- claim, race conditions) — not acceptable (display a slightly stale counter).

Architectural responses

  1. Migrate the allocation state to a transactional store (patterns/transactional-db-over-eventually-consistent-kv-for-claim). In Browser Run's migration: KV → D1, with a UPDATE...WHERE...IN (SELECT...) RETURNING pattern that atomically claims and returns in one statement (concepts/sqlite-transaction-for-atomic-resource-claim).
  2. Keep eventually-consistent state for the parts that tolerate it (e.g. observability, dashboards, cross-region capacity overview). The allocation hot path migrates; the global-view aggregation can stay on KV.
  3. Bound the per-shard rate at a level the eventually- consistent store can handle. Often viable for low-volume admin operations; rarely viable for an allocation hot path subject to demand spikes.

When the failure mode does not apply

  • Idempotent claims — if the action is idempotent (the same resource being "claimed" twice is fine), the eventually-consistent read is harmless.
  • Sticky routing eliminates the race — if the same caller is always routed to the same shard, claims happen on a single process and the eventual-consistency floor is irrelevant.
  • Lease semantics absorb the gap — if claims auto-expire faster than the store's T, an over-claim is bounded (Slicer, Centrifuge style). Browser Run could not adopt this because the resource (a browser instance) is too expensive to lease-and-discard at every tick.

Seen in

Last updated · 542 distilled / 1,571 read