Skip to content

CONCEPT Cited by 1 source

Distributed log

Definition

A distributed log is an ordered, append-only sequence of records distributed across multiple nodes, addressed by a monotonically increasing offset, with:

  • O(1) writes at the tail and O(1) reads at the head or tail (so throughput is independent of log size),
  • Immutability (records never modified in place) — efficient for concurrent reads without locking,
  • Sequential byte-layout that matches the access pattern HDDs and filesystems are fastest at (see concepts/hdd-sequential-io-optimization),
  • A replication protocol that gives multiple nodes the same log state.

The canonical production instance is Apache Kafka's topic-partition — each partition is a log; replicas of the partition are copies of the log on different brokers.

Why it matters

Kozlovski's Kafka-101 framing:

"The data in the system is stored in topics. The fundamental basis of a topic is the log — a simple ordered data structure which stores records sequentially. […] It's immutable and has O(1) writes and reads (as long as they're from the tail or head). Therefore the speed of accessing its data doesn't degrade the larger the log gets and, due to its immutability, it's efficient for concurrent reads. But despite these benefits, the key benefit of the log and perhaps the chief reason it was chosen for Kafka is because it is optimized for HDDs." (Source: sources/2024-05-09-highscalability-kafka-101)

Two cross-cutting design levers fall out:

  • Decoupling producers and consumers — unlike a traditional message bus that deletes on consume, a log retains past records so many independent consumer groups can read the same topic at their own pace. This was the design choice that let Kafka beat the message-bus generation of systems.
  • Replayability — consumers that crash can resume from the last committed offset; new consumers can start from the beginning.

Relationship to other log primitives

  • Database WAL — a log used internally for durability of a single database; not typically exposed as a first-class data-plane abstraction.
  • Litestream LTX log (concepts/ltx-file-format) — a replication-log transport for SQLite, log-as-wire-format.
  • Distributed log (this page) — the log is the API; producers write to it, consumers read from it, replication and ordering are first-class promises.

Seen in

Last updated · 319 distilled / 1,201 read