Skip to content

CONCEPT Cited by 2 sources

Log compaction

Definition

Log compaction is a retention strategy in append-only distributed logs (canonically Apache Kafka) where the broker retains only the latest value for each key in a topic, rather than retaining records by time or size. Compaction transforms an unbounded event stream into a bounded key-value changelog — the compacted log represents the latest state of every key ever written.

Mechanism

In Kafka, a background log cleaner thread periodically scans closed log segments and removes older records that have been superseded by a newer record with the same key. Two special record types interact with compaction:

  1. Tombstones — records with key K and null value. They signal deletion of key K. After compaction removes all prior values for K, the tombstone itself becomes removable after delete.retention.ms (default 24 h).

  2. Transaction control batches — COMMIT/ABORT markers appended by transactional producers. These are subject to the same expiration-based cleanup: once resolved data is compacted away, the markers become removable.

The compaction–replication race (Kafka 3.9–4.2)

In Kafka, each broker compacts its own replica independently — there is no coordination between replicas about compaction progress. This creates a race condition: if a critical metadata record (tombstone or control batch) is written while a replica is offline and compacted away before that replica catches up, the replica permanently diverges from the leader.

"Tombstones and COMMIT/ABORT control batches are the only signals that their associated records were deleted, committed, or aborted, respectively. Once a tombstone or a control batch is compacted away, this information is gone." (Source: sources/2026-06-25-redpanda-kafkas-log-compaction-corrupts-data)

See concepts/compaction-replication-race for the full four-variant bug taxonomy and concepts/coordinated-compaction for Redpanda's fix.

The memory problem and SHA-256 hashing

The log may contain an unbounded number of unique keys, but the key-offset map must fit in finite memory. Redpanda uses a SHA-256 hash of each key (32 bytes) paired with an 8-byte offset = fixed 40 bytes per entry. With a default allocation of 128 MiB, this fits ~3.3 million keys per compaction pass.

Two optimizations ensure convergence to full deduplication over finite passes:

  1. Clean-range bookkeeping — ranges already compacted need not be re-indexed in future passes.
  2. Backward scanning of dirty ranges — scanning dirty segments in reverse guarantees that the first offset indexed for any key is the latest. The scan can stop early when the map is full.

(Source: sources/2026-06-30-redpanda-cloud-topics-compaction)

Redundant compaction in disk-based Kafka

In traditional disk-based Kafka (and Redpanda's NVMe-backed topics), each broker compacts its own replica independently. With replication.factor=3, the same logical data is compacted 3 times on 3 brokers, wasting CPU cycles that could serve produce/consume traffic. Compression further amplifies the cost: each rewrite requires decompression and recompression.

(Source: sources/2026-06-30-redpanda-cloud-topics-compaction)

Cloud Topics: single-copy compaction on object storage

Cloud Topics eliminates redundant compaction entirely. Because committed data lives once in object storage (as immutable L1 objects) and metadata is tracked in the metastore, compaction:

  • Runs once against the canonical copy
  • Can execute on any shard on any node (not just the partition leader)
  • Uses a pull-based scheduler with a dirty-ratio + lag priority queue
  • Uploads results via multi-part upload, capping memory at part size
  • Achieves optimistic concurrency via a compaction_epoch integer — stale writers are rejected atomically

This also eliminates the tombstone coordination problem: with a single copy, there are no replicas to diverge.

(Source: sources/2026-06-30-redpanda-cloud-topics-compaction)

Seen in

  • sources/2026-06-25-redpanda-kafkas-log-compaction-corrupts-data — canonical source disclosing Kafka's uncoordinated compaction as a data-corruption vector. Four failure modes demonstrated: deleted data reappears, aborted data served as committed, committed data hidden, partition frozen for read_committed consumers. Redpanda's coordinated compaction protocol introduced as the fix.
  • sources/2026-06-30-redpanda-cloud-topics-compaction — explains how Cloud Topics rethinks compaction: single-copy on object store, pull-based scheduling, SHA-256 key hashing, multi-part upload, and optimistic concurrency via compaction_epoch.
Last updated · 609 distilled / 1,932 read