Skip to content

PATTERN Cited by 1 source

In-place partitioned-to-clustered conversion

Problem

Migrating a partitioned table to Liquid Clustering historically required a full table rewrite plus a consumer-cutover dance:

  • REPLACE TABLE rebuilds the entire table with the new layout, but produces a downstream-breaking change (table identity may change; downstream consumers fail until they're updated).
  • Dual writes + planned downtime cutover — write to the old and new tables in parallel; backfill the gap; cut over consumers; tear down the old table. Operationally complex; requires coordinated changes across many pipelines; pays full-rewrite compute cost.

For tables hosting continuous ingestion under regulatory constraints (CDC tables, audit logs, financial records) the "planned downtime" is a non-starter — ingestion cannot be paused during the cutover.

The 2026-06-01 Databricks "Debunking 8 data layout myths" post:

"Before, converting a partitioned table to Liquid Clustering required a full table rewrite and downstream breaking changes with REPLACE TABLE or a cutover with dual writes and planned downtime."

Solution

Convert in place via a single SQL command that preserves table identity, runs alongside live ingestion, and minimises rewrites. The 2026-06-01 source discloses the SQL surface (Private Preview as of 2026-06-01):

ALTER TABLE my_table REPLACE PARTITIONED BY WITH CLUSTER BY (col1, col2, ...)

The 2026-06-01 source's verbatim framing of the new command:

"We're introducing a new command (now in Private Preview) that makes this conversion easier, minimizing both downtime and rewrites."

Structural pieces

Piece What it does
Same-table identity Table name, location, downstream references unchanged. Consumers see the same table; only physical layout evolves.
Live-ingestion compatibility Conversion proceeds while writes continue; new writes follow the new clustering keys; old data is gradually re-clustered.
Minimised rewrites Existing partition directories aren't full-rewritten; the engine reorganises files lazily as part of normal maintenance.
No downstream breaking changes No REPLACE TABLE; no consumer cutover; no schema break.

The Bolt CDC validation

The 2026-06-01 source's primary case study for the in-place conversion is Bolt's CDC table migration (TB-scale):

"Bolt recently tried Liquid Conversion (currently in Private Preview), which converts partitioned tables to Liquid in-place using ALTER TABLE .. REPLACE PARTITIONED BY WITH CLUSTER BY. They observed the following read and write benefits on a TB-scale CDC table after converting to Liquid Clustering: - Write throughput (rows/sec) increased by 138% - Read times were reduced by up to 63%, with an average of 21% reduction across 9 representative queries"

Verbatim from Bolt's senior platform engineer:

"Liquid Clustering dramatically reduced the work that each write was doing, increasing our throughput significantly on our most critical CDC table. Reads also improved across the board. The best thing was: we ran the conversion from partitioning alongside live ingestion with zero downtime. With this, Liquid Clustering provided us exactly the kind of performance and reliability we needed at platform scale."

The zero-downtime + live-ingestion property is the load-bearing operational claim. CDC tables specifically can't tolerate cutover downtime; the in-place conversion makes the migration safely performable on this workload class.

Why this matters

Removes the operational tax on migration

Pre-Liquid-Conversion, the partition→Liquid migration paid an operational tax that often outweighed the steady-state benefit:

  • Engineering time on cutover coordination
  • Compute cost of full table rewrite
  • Risk of cutover failures
  • Required-downtime negotiations with stakeholders

Many teams left tables partitioned despite knowing over-partitioning was hurting them — the cost of migrating exceeded the per-quarter pain of putting up with the partition. In-place conversion changes that calculus: the migration is a single SQL command run alongside live writes.

Removes the "we can't change layout once chosen" architectural

ceiling

The 2026-06-01 source canonicalises this as the broader architectural payoff:

"In 2026, the layout should be an implementation detail of the table, with every engine that reads or writes benefitting from it."

In-place conversion is the operational mechanism that delivers this ceiling: layout choices are reversible at any time, with no downtime, no rewrite, no downstream changes. The architectural abstraction "layout is implementation detail" requires operational tooling that makes layout changes cheap; in-place conversion provides that tooling.

Composition with patterns/clustering-keys-as-engine-input

The pattern composes with the broader inversion: clustering keys are pure intent declarations; layout is engine-owned. In-place conversion lets the architect change the intent declaration at any time, with the engine then propagating the layout change through incremental maintenance.

In effect: the partition-vs-cluster decision becomes non-load-bearing. Whatever choice was made at table creation can be revisited as the workload evolves, without consequences for downstream consumers or operations.

Failure modes (inferred — not disclosed in source)

  • Partition columns embedded in semantic. If downstream tools treat the partition column as a logical property of the table (filter on it, group by it, expect it to be in the directory path), the conversion may be transparent at the storage layer but break tooling expectations. Mitigation: audit consumers before conversion.
  • Conversion rate vs. ingest rate. If ingest throughput exceeds the engine's incremental-conversion bandwidth, the table can remain in a mixed state for an extended period. Mitigation: source doesn't disclose; presumably the engine sequences the work to converge.
  • Existing partition directory layout reuse. Source doesn't disclose whether existing partition directories are repurposed in place or new files are written and the old eventually reclaimed; the latter implies temporary 2× storage.

Sibling migration patterns

  • patterns/shadow-then-reverse-shadow-migration — the classical migration pattern when the target storage system can't support in-place transitions; pays the dual-write tax.
  • REPLACE TABLE — full-rewrite-and-replace; pays both the rewrite tax and the downstream-break tax.
  • Dual writes + cutover — pre-Liquid-Conversion approach; pays the operational coordination tax.

The pattern this page describes is the first canonical disclosure on the wiki of an in-place lakehouse layout conversion — most prior storage-layout migrations on the wiki are full-rewrites or shadow migrations.

Seen in

  • sources/2026-06-01-databricks-debunking-8-data-layout-myths-why-liquid-clustering-outperfo — First wiki canonicalisation as a named pattern. Bolt's TB-scale CDC table migration is the operational evidence: +138% write throughput, −21% avg / −63% max read time, zero downtime alongside live ingestion. Reserved for future ingests: GA timeline, conversion-time write-amplification envelope, behaviour on tables with foreign-key-style partition designs, multi-region consistency during conversion, the conversion progress / state-tracking surface.
Last updated · 542 distilled / 1,571 read