Skip to content

SYSTEM Cited by 1 source

NeonVM

NeonVM is a custom Kubernetes resource and controller — built on QEMU and KVM — that can add or remove CPU and memory from a running guest VM without stopping it, and can live-migrate a VM between nodes while preserving its IP address. It is the mechanism that executes Lakebase Postgres compute resizes decided by the autoscaling algorithm.

NeonVM originated at Neon before the Databricks acquisition; per the autoscaling post, "Lakebase Postgres architecture started in Neon and the resource/controller name remains the same."

Why a VM (not a container)

Each Postgres instance in Lakebase runs inside its own virtual machine in a Kubernetes cluster. The post's stated reasons:

  • Strong isolation boundary between tenant databases.
  • Unlike a conventional container allocation, a VM allows CPU and memory to be added to or removed from a running guest — the core requirement for in-place vertical autoscaling.

Role in the resize control loop

NeonVM is one of four coordinating components (Source: sources/2026-08-31-databricks-autoscaling-lakebase-postgres):

  1. autoscaler-agent (per Kubernetes node) — collects metrics, computes the target size, initiates scaling.
  2. vm-monitor (inside each VM) — watches Postgres memory at 100ms cadence, validates downscale requests, resizes the compute cache.
  3. Modified Kubernetes scheduler — the single source of truth for allocation; every upscale must be approved before memory is committed.
  4. NeonVM — the custom resource + controller that applies the change to the running VM.

Scale-up path

agent computes target → scheduler approves (no memory overcommit) → agent updates the NeonVM resource → NeonVM controller adds CPU/memory to the running VM → vm-monitor expands the compute cache (LFC).

Live migration fallback

When a node is too full to grow a VM in place, NeonVM live-migrates the VM to another node. The VM keeps its IP address, so existing connections stay open. Because Lakebase compute nodes hold little durable local state (storage/compute separation), a migration is mostly moving VM memory and runtime state — not the database itself, which lives in pageservers and safekeepers.

Relationship to the Kubernetes operator pattern

NeonVM is a concrete instance of the operator pattern: a custom resource (the desired VM shape, including CPU/memory) plus a controller that reconciles a running QEMU/KVM guest toward that desired state. The autoscaler-agent drives the desired state by editing the NeonVM resource.

What's undisclosed

  • The CU→(vCPU, RAM) mapping and the granularity/step size of in-place resize.
  • Whether the compute cache survives a live migration or re-warms on the destination node.
  • The QEMU/KVM mechanism used for hot CPU/memory add-remove (e.g. memory ballooning vs hotplug) is not named.

Seen in

Last updated · 766 distilled / 2,225 read