Skip to content

CONCEPT Cited by 1 source

RocksDB TTL compaction

Definition

A state expiry mechanism where expired keys are removed during RocksDB's background compaction process rather than through explicit per-element timer registration. The TTL (time-to-live) is encoded in the key/value metadata; when a compaction pass encounters an entry whose TTL has elapsed, it discards the entry.

Why it matters for streaming

In stream processors like Flink, TTL compaction provides an alternative to explicit timer-based state cleanup. For workloads with high key cardinality and bounded retention windows:

  • No per-element timer overhead — avoids maintaining a sorted timer queue with millions of entries
  • Asynchronous cleanup — happens in background threads during compaction, not on the hot path
  • Amortised cost — cleanup batched with normal LSM-tree maintenance

Tuning requirements

Default RocksDB settings may be too conservative for streaming workloads where high write rates produce many short-lived keys. Without tuning, compaction runs too infrequently to exercise the TTL filter, causing unbounded state growth.

Key tuning parameters (from Zalando's production configuration): - Write buffer count: 16 (default 2) - Write buffer size: 512 MB (default 64 MB) - Compaction target file size base: 256 MB (default 64 MB) - Level max size base: 1 GB (default 256 MB) - Compression: LZ4 (default Snappy) - Predefined options: FLASH_SSD_OPTIMIZED - Post-rescale: async compaction + range delete for orphaned keys

Seen in

Trade-offs

  • Pro: Zero per-element timer cost, background processing, batched cleanup
  • Con: Non-deterministic expiry timing (depends on compaction schedule), requires careful tuning of compaction frequency vs write amplification
  • Caveat: Only works with incremental checkpoints — not compatible with all Flink cleanup strategies
Last updated · 595 distilled / 1,807 read