Skip to content

SYSTEM Cited by 1 source

AutoLiquid (autonomic data-layout optimization)

AutoLiquid is Databricks' system for autonomic data-layout optimization across the Lakehouse: it automatically selects and maintains clustering keys for tables so that queries prune better and scan less. It is the intelligence behind the single CLUSTER BY AUTO primitive on Liquid Clustering tables. Published as a VLDB 2026 paper (presenter: Yunjia Zhang). (Source: sources/2026-08-27-databricks-building-for-the-ai-era-lakebase-streaming-and-lakehouse-innovations-vldb-2026)

Problem

Clustering a table by the right keys can dramatically improve query performance (better file-level data skipping), but manually choosing optimal clustering keys does not scale across millions of Lakehouse tables — the number of tables, the diversity of query predicates, and workload drift over time make hand-tuning intractable. This is the fleet-scale version of the problem that Liquid Clustering solves per-table.

How it works

  1. Heuristic key selection. AutoLiquid uses a combination of heuristics to propose clustering keys for a table based on its query/access patterns.
  2. Shadow verification. Before adopting a candidate layout, AutoLiquid validates it efficiently on a shadow copy rather than disrupting the live table — the shadow-verification pattern that lets the system self-tune safely at scale.
  3. Fleet-scale application. With cheap verification, AutoLiquid clusters millions of tables autonomically, exposed to users as CLUSTER BY AUTO.

Results

  • Outperforms customer-selected clustering keys on over 95% of evaluated workloads — i.e. the automated choice beats the human choice in the vast majority of cases. (Vendor-reported.)

Relationship to other systems

  • Sits on top of Liquid Clustering — AutoLiquid chooses the keys; Liquid Clustering executes the co-location without fixed partition boundaries.
  • Part of the broader Predictive Optimization / automatic table optimization story, where the platform maintains data layout on the user's behalf.

Seen in

Last updated · 766 distilled / 2,225 read