Skip to content

PATTERN Cited by 1 source

Usage-driven migration prioritisation

Problem

Migrating a large analytics-table fleet to a new storage / table / catalog format (Apache Iceberg in Yelp's case) is a multi-quarter backlog with a long tail of tables. Migrating in arbitrary order — alphabetical, by team ownership, by historical sequence — risks delivering migration benefits to the lowest-value tables first: a table no one reads gets the latency improvements of the new format, while the busiest table still suffers the legacy format's costs.

The structural failure mode: migration effort is spent on tables that no consumer notices, while consumer-facing latency wins land late or never.

Pattern

Use granular usage attribution data to prioritise migration toward the most-accessed tables and partitions first. Same observability substrate that powers cost-optimisation decisions and stakeholder discovery also ranks the migration backlog by active-table value.

The structural insight: a single observability investment amortises across multiple platform-team initiatives — the same (table, partition, iam_role, event_time) aggregate serves cost attribution, retention policy, AND migration prioritisation.

Canonical shape (Yelp 2026-05-21)

"We set out to identify active tables and partitions to prioritize our adoption of Apache Iceberg, but we soon realized that the opportunity was larger: partition-level usage data would deliver meaningful improvements across the three main pillars of a platform team."

The headline framing: Iceberg migration prioritisation was the original motivation; the storage-class / Default-Access-Retention / 33%-cost-reduction wins emerged as a second-order effect of the same observability data.

1. Identify active tables and partitions

Aggregate access events from compacted S3 SAL to compute per-table and per-partition access counts over a recent window (per patterns/access-pattern-visualization-for-data-stewardship).

2. Rank the migration backlog

Sort tables descending by:

  • Total access count — most-accessed tables first.
  • Distinct consumer count (number of distinct IAM roles reading) — broadly-consumed tables first.
  • Recency-weighted access — tables with active recent consumption first, even if historical totals are smaller.

3. Migrate top-of-list tables first

"By identifying the most frequently accessed data, the data platform team was able to focus our migration efforts on active tables and partitions that would add the most customer value. This enabled the team to provide Apache Iceberg's read performance benefits to the most valuable use cases first."

4. Migrate hot partitions before cold partitions within a table

For partitioned tables, the same logic applies at partition grain: the most-recently-accessed partitions migrate first. Cold partitions that are candidates for Default Access Retention may be deprioritised entirely — if the partition is gated behind IAM, the migration value is lower (few consumers, no read-performance wins to deliver).

(Source: sources/2026-05-21-yelp-how-partition-access-visualizations-reduced-our-data-lake-s3-cost-by-33)

Forces that make the pattern work

  1. Consumer-facing wins land first. Latency improvements from the new format reach the highest-value consumers before low- value consumers.
  2. Usage data is already produced for cost attribution and stakeholder discovery — incremental cost is near-zero.
  3. The same chart enables data owners to participate. A data owner reading the partition-access-pattern chart can immediately see why their table is or isn't on the migration backlog and advocate for prioritisation.
  4. Migration backlogs naturally have a long tail. The 80/20 of access typically lives in 20% of tables; prioritising those 20% delivers most of the value at fraction of the effort.

Forces that limit the pattern

  1. Schema-complex tables may take disproportionate effort. A simple low-traffic table may migrate faster than a complex high-traffic one; usage-only ranking ignores effort.
  2. Migration risk varies. Some tables have rigid downstream contracts; usage-only ranking ignores migration safety risk.
  3. Cold tables that are operationally important may be under-prioritised. Compliance archives, regulatory data — low access count, high importance.

In practice, usage-driven ranking is one of multiple inputs to the prioritisation decision; it gives the operator a strong default to deviate from rather than a single sort key.

Generalisation beyond Iceberg

The pattern applies to any large-scale fleet migration:

  • Storage-format migration (Parquet → Iceberg, Delta, Hudi).
  • Compute-engine migration (legacy MR → Spark, Spark → Photon).
  • Catalog migration (Hive Metastore → Unity Catalog, Glue → Polaris).
  • Schema migration (legacy denormalised → normalised, v1 → v2).

In all cases, the same observability substrate that drives cost- optimisation decisions can drive migration prioritisation.

Outcomes (qualitative, no numbers)

The Yelp post does not disclose:

  • Number of tables migrated.
  • Migration speed-up vs unprioritised counterfactual.
  • Latency improvements at table grain.
  • Migration backlog size before / after.

The qualitative claim is "provide Apache Iceberg's read performance benefits to the most valuable use cases first" — strategy disclosure rather than measured outcome.

Seen in

Last updated · 542 distilled / 1,571 read