Skip to content

PATTERN Cited by 1 source

Separate data-plane controller for hot-path operations

The pattern

When a subset of the control plane's verbs runs on the synchronous request path of customer requests (because of agentic / serverless / scale-to-zero workload shape), decompose the control plane into two services:

  1. A data-plane controller that handles only the hot-path verbs (e.g. database start / suspend / connection routing). Designed with less business logic, a strict minimal external dependency set, defense-in-depth, and graceful-degradation discipline.
  2. A management control plane that handles cold-path verbs (billing, governance, key rotation, schema management, asset inventory). Operates on traditional control-plane availability discipline.

The pattern is the architectural answer to "control plane is the new data plane" — the inversion-corner-case where one verb's availability requirements are at data-plane tier even though the verb structurally lives in the control plane.

Why not just merge the planes

The naive merge — "if start is data-plane-tier, put it back in the data plane" — re-introduces the very problems control / data plane separation solves: scaling profiles diverge, blast radius recombines, language / framework choices conflict. The pattern preserves the architectural separation but carves out a hot-path-only sub-service from the control plane with stricter availability discipline and minimal dependencies.

Lakebase canonical framing

Verbatim (Source: sources/2026-05-27-databricks-how-the-lakebase-architecture-stays-resilient-to-cloud-failures):

"Currently, our control plane handles everything from starting databases to billing. The former is clearly more critical. We've had outages where background maintenance operations resource-starved on-demand database startups - that's clearly not ok. We're currently hard at work separating the critical parts of the control plane into a data plane controller service that handles only hot-path operations (start/suspend). This service has less business logic, a strict, minimal set of external dependencies, and is engineered from the ground up with resilience, graceful degradation, and defense-in-depth top of mind."

The three named design properties of the data-plane controller:

  1. Less business logic — the service does one thing (start / suspend) and does it with minimum complexity. Billing rules, governance, key rotation, etc. live in the management control plane and are not on the start path.
  2. Strict minimal external dependencies — pairs with concepts/critical-path-dependency-minimization + patterns/preallocated-bare-metal-pool-with-virtualization to remove the cloud-provider control-plane chain from the start path.
  3. Defense-in-depth + graceful degradation discipline — the service must keep operating under partial failures of its small dependency set; degrade rather than fail closed.

The trigger anti-pattern

The Lakebase post discloses the failure mode that motivated the split:

"We've had outages where background maintenance operations resource-starved on-demand database startups - that's clearly not ok."

This is a shared-fate-via-shared-runtime anti-pattern: hot-path start operations and cold-path background maintenance ran on the same control-plane runtime, so a runaway maintenance job consumed the resources the hot path needed. Single-runtime SLA classes (thread-priority / scheduler weighting) only go so far; the structural fix is separate runtimes.

Where to draw the boundary

The boundary is per-verb, not per-system. Verbs that go in the data-plane controller:

  • Start / resume database (auto-resume from idle, cold-create on agent provisioning).
  • Suspend database (idle scale-to-zero — affects billing but is hot-path because customer-visible).
  • Route connection (in some architectures the proxy / connection-router lives here; in others it's a separate data plane).

Verbs that stay in the management control plane:

  • Billing (cold-path; eventual is fine).
  • Schema management / migrations (customer-initiated, not per-request).
  • Governance / key rotation / IAM updates (ops-initiated).
  • Asset inventory / audit log emission (background, eventual).
  • Capacity planning / pool replenishment (background; replenishes the pool the hot-path controller draws from).

Composability

The pattern composes naturally with several others:

Caveats

  • Boundary maintenance is ongoing. New control-plane verbs are added over time; the discipline of asking "is this hot-path?" before deciding which side to put them on is a perpetual cost.
  • Code duplication risk. The two services may share business- logic libraries; if those drift, behaviour drifts. (See the Aurora DSQL contradiction on language-per-plane in concepts/control-plane-data-plane-separation — the same drift-risk applies even within one language.)
  • Operational discipline must follow the architectural split. If on-call for both services is the same team and the same runbook, the structural separation doesn't help during an incident.
  • Migration is hard. Splitting an existing single-control-plane service into hot-path + cold-path requires identifying the hot-path verbs, untangling shared state, and rolling out without customer impact. Lakebase calls this "currently hard at work" — i.e. in flight, not done.

Seen in

Last updated · 542 distilled / 1,571 read