Skip to content

DATABRICKS 2026-10-13

Read original ↗

What is HTAP? A plain-English guide to hybrid transactional/analytical processing

Summary

Databricks' plain-English guide defines HTAP (Hybrid Transactional/Analytical Processing) — a term Gartner coined in 2014 — as a database architecture that brings OLTP and OLAP closer together so applications can run analytical queries on fresh operational data without first moving it through a batch ETL pipeline into a separate warehouse. The guide's load-bearing distinction is converged vs composed HTAP: a converged engine runs both workloads through a single database engine (sharing much of the storage, possibly with different indexes/formats/execution paths), while a composed architecture uses separate transactional and analytical engines inside one integrated system, coordinated through a shared storage layer or change propagation. Both face the same core tension — give analytics fast access to operational data without compromising transaction latency and reliability. The post enumerates the four recurring HTAP challenges (write amplification, resource contention, consistency/freshness, operational complexity), gives a six-criterion evaluation checklist, and walks three real systems — TiDB (composed: TiKV rows + TiFlash columnar, Raft-replicated, snapshot-isolation version-checked reads), AlloyDB (converged: transparent in-memory columnar engine on Postgres, read-pool instances, Google's claimed 100× analytical speedup), and Databricks Lakebase + LTAP (separate compute over a shared data layer). It closes with a when-to-use decision frame and a six-step incremental migration playbook, and self-servingly positions LTAP as the Databricks answer.

Key takeaways

  1. HTAP = analytics on current operational data without the ETL hop. Gartner coined the term in 2014; the pitch is that decisions get made "on the latest transactions rather than on yesterday's extract." For many operational- analytics use cases HTAP can reduce or remove ETL, cut the number of data copies, and move toward a single source of truth for both operational and analytical systems. (Source: this article)

  2. Converged vs composed is the architectural fork. Converged HTAP runs transactional and analytical workloads through one database engine (they may use different indexes, storage formats, or execution paths but share much of the same database). Composed HTAP uses separate transactional and analytical engines inside an integrated architecture, sharing a storage layer or propagating changes via replication while each stays optimized for its workload. Trade-off: a converged engine avoids synchronizing independent query engines but must stop analytics from interfering with latency-sensitive transactions; a composed architecture gets freedom to optimize/isolate each workload but must coordinate data versions and keep the analytical path fresh. (Source: this article)

  3. Why OLTP and OLAP split in the first place is storage physics. Transactional queries read/update a few complete rows → row-based storage, selective indexes, millisecond latency. Analytical queries scan millions of rows across a subset of columns → columnar storage for compression and less wasted I/O, tolerating seconds of latency. Optimizing them separately made sense but "builds latency into every analytical answer" via the data-movement delay; the disaggregated stack (CDC feeds, stream processors, cloud warehouses/lakes, real-time query layers) grew to close that gap, each hop adding cost and another place consistency can break. (Source: this article)

  4. An HTAP architecture has four generic parts. (1) a transactional path — row-oriented storage, indexes, caches, a write-ahead log for low-latency durable writes; (2) an analytical path — columnar representations, possibly on separate compute, often with an intelligent query optimizer that decides whether a query reads the row store, the column store, or both; (3) storage / data representation — frequently two representations of the same data (row engine + column engine), within one system or across components, sometimes decoupling compute from object storage; (4) synchronization & freshness — when the paths use different engines, committed changes propagate via CDC, WAL replication, shared logs, or shared storage. (Source: this article)

  5. Four recurring HTAP challenges. (a) Write amplification — storing data in two formats means updates generate deltas or require refresh/compaction, adding compute/I/O/storage even when kept off the transaction path. (b) Resource contention — when OLTP and OLAP share resources, large analytical queries can push transaction latency "from milliseconds to seconds"; separate compute helps but shared storage/network/coordination remain bottlenecks. (c) Consistency & freshness — the two paths may not advance at the same rate; greater freshness needs tighter coordination, and "every HTAP architecture manages some tension between synchronization and isolation rather than eliminating it." (d) Operational complexity — multiple representations, execution paths, resource policies, sync mechanisms; the optimizer must plan across two formats. (Source: this article)

  6. TiDB is the canonical composed-HTAP example. (2020 VLDB paper: a Raft-based HTAP database.) Transactional workloads use TiKV (row store); TiFlash maintains a columnar replica for analytics. TiDB's optimizer routes a query to TiKV, TiFlash, or both, with the two paths on separate nodes for workload isolation. TiFlash receives replicas asynchronously via the Raft log but checks whether it has reached the data version a query requires to provide snapshot isolation. Evaluation watch-out: replication lag's effect on analytical latency/throughput under sustained writes. (Source: this article)

  7. AlloyDB is the converged example; Lakebase/LTAP the shared-storage split. AlloyDB runs a transparent in-memory columnar engine on Postgres that accelerates scans/joins/aggregations, on the primary or on read-pool instances for isolation; Google claims up to 100× faster analytical queries than standard PostgreSQL. Watch-outs: columnar coverage (unsupported ops or data not in the columnar engine fall back to the row path) and refresh cost under frequent updates. Lakebase with LTAP goes further — it keeps transactional (Postgres) and analytical compute independent, connected through a shared data layer: analytical compute reads the bulk from a columnar copy in object storage, fetches only the current log position from Postgres, and merges recent changes from the pageserver for a transactionally consistent view without placing analytical load on the primary. (Source: this article)

  8. When HTAP is the right call, and when to keep OLTP/OLAP separate. Use HTAP when "the value of your analysis depends on access to fresh operational data" (real-time BI, operational analytics, AI agents, feature serving, fraud detection) — start by asking how much delay the workload tolerates. Keep them separate when analytics can tolerate delay (historical analysis, model training, regulatory reporting) or when existing CDC/ETL already meets freshness/perf/cost — then HTAP adds migration risk and complexity for little benefit. (Source: this article)

Systems / concepts / patterns extracted

  • Systems: systems/tidb (composed-HTAP architecture described in substance — TiKV/TiFlash/Raft/version-checked snapshot reads), systems/lakebase (shared-data-layer separation via LTAP), systems/postgresql (the engine AlloyDB and Lakebase both extend). AlloyDB named with a section's worth of detail but recorded as a frontmatter slug + plain-text prose only — NO page, NO link (Systems-gate rule 3/4: single source, not the article's primary subject; this is a vendor-neutral survey, not an AlloyDB architecture post).
  • Concepts: concepts/oltp-vs-olap (the page that owns the htap alias — this is HTAP's canonical home), concepts/ltap (the sibling architecture the post contrasts and promotes), concepts/columnar-storage-format, concepts/change-data-capture, concepts/wal-write-ahead-logging, concepts/write-amplification, concepts/noisy-neighbor (resource contention), concepts/snapshot-isolation (TiFlash version-checked reads), concepts/compute-storage-separation.
  • Patterns: none minted. converged-HTAP and composed-HTAP are the two HTAP sub-shapes; they live as prose + tags on the HTAP-owning concepts/oltp-vs-olap page, not as standalone pattern pages (they only mean anything as a decomposition of HTAP, which is already canonicalized).
  • Named-but-not-minted (tags/prose per taxonomy gate — single-source or vendor-specific): converged vs composed HTAP; in-memory transparent columnar engine; read-pool instances; optimizer routes to row-store / column-store / both; replication-lag-vs-analytical-freshness; the six-step migration playbook. If a second source describes "converged vs composed HTAP" as a reusable taxonomy, Lint could promote it onto the HTAP page as a formal sub-section (it already has a home there).

Operational numbers

  • AlloyDB: Google-stated up to 100× faster analytical queries than standard PostgreSQL (vendor claim, not independently benchmarked here).
  • Resource contention generic figure: large analytical queries can push OLTP transaction latency "from milliseconds to seconds" under shared resources.
  • Gartner coined HTAP in 2014; TiDB's HTAP design published at VLDB 2020.
  • No first-party throughput/latency numbers for the surveyed systems beyond the AlloyDB vendor claim — this is a conceptual guide, not a benchmark post.

Evaluation checklist (verbatim criteria)

The post's six-criterion test, to run with transactional + analytical workloads together under production-like conditions:

Criterion What to test
OLTP p99 latency under mixed load p99 transaction latency stays within target as analytical load rises
Analytical performance at scale scan/aggregation perf as data volume, query complexity, concurrency grow
Freshness and consistency how fast committed changes become queryable; guarantees when the analytical path falls behind
Cost under mixed load compute, storage, caching, replication, network, data-transfer costs at representative traffic
Failure and recovery query behavior during failures, correctness under degraded conditions, backlog growth, recovery speed
Operational overhead tuning, monitoring, capacity planning, workload management, incident response effort

Six-step migration playbook: (1) map workloads/dependencies and separate truly both-needing ones; (2) set guardrails (latency/perf/freshness/correctness/ availability/cost thresholds + rollback triggers); (3) start with analytical reads in parallel with the existing path; (4) shift traffic gradually; (5) test failure and recovery before depending on it; (6) migrate writes only if the system of record changes — validate durability/schema-evolution/downstream/ rollback first.

Caveats / limitations

  • Vendor-authored survey with a destination. It's a Databricks post whose conclusion steers to LTAP/Lakebase; treat the comparative framing (converged/composed vs LTAP) as Databricks' positioning, not neutral benchmarking. The TiDB and AlloyDB descriptions are, however, accurate at the architecture-sketch level.
  • No independent numbers. The only quantitative claim is Google's 100× AlloyDB figure; nothing is measured in-post.
  • "HTAP has been pitched as the next big DB architecture more than once" — the post itself flags HTAP's history of over-promising and is careful to recommend it only when fresh-operational-data value is the actual constraint.
  • The guide is conceptual (definitions, FAQs, decision frames); it names TiDB, AlloyDB, and Lakebase as illustrations rather than documenting any of them in implementation depth.

Source

Last updated · 854 distilled / 2,448 read