Skip to content

SYSTEM Cited by 1 source

Redis Streams

Redis Streams is a Redis 5.0+ data type modelled as an append- only log with consumer-group semantics — the closest in-Redis analogue to Kafka topics. Producers write with XADD; consumers read with XREADGROUP (consumer-group flavour) or XREAD (single- reader flavour). Each entry has a server-generated monotonically increasing ID and an arbitrary key/value payload.

Within Redis it is the only built-in primitive with persistent ordered delivery + consumer groups + replay-from-offset semantics — pub/sub is fire-and-forget, lists are queues, sorted sets are not delivery-shaped. Redis Streams sits at the "small-scale broker" sweet spot: durable, ordered, multi-consumer, but without Kafka's partition-level parallelism, durable replay across long windows, topic-introspection tooling, or programmatic provisioning APIs.

Limits at scale

The canonical wiki disclosure of Redis Streams' scale ceiling is the OmniNode → Redpanda migration. The migration trigger was not throughput — at OmniNode's scale Redis Streams handled the message rates fine. The trigger was coordination capabilities the Kafka model offers that Redis Streams does not:

  • Consumer groups that work the way Kafka's do (partition- level parallelism with at-most-once-per-partition consumer assignment).
  • Partition-level parallelism for fan-out across many workers.
  • Durable replay semantics at the offset granularity Kafka offers.
  • Topic introspection tooling.
  • Programmatic provisioning APIs.

Quote: "We outgrew Redis Streams not because of throughput, but because coordination itself became difficult."

Seen in

  • sources/2026-06-02-redpanda-how-omninode-uses-redpanda-to-scale-ai-agent-workflows (2026-06-02, OmniNode founder Jonah Gray on Redpanda Blog) — canonical disclosure of Redis Streams' scale ceiling. OmniNode used Redis Streams behind a transport-layer abstraction ("published with XADD, consumed with XREADGROUP, and kept topic names in Python constants near the code that used them. Apache Kafka was explicitly deferred in the roadmap because the system was still small.") until 5 → 12 repos / 100+ event types made consumer-group / partition-parallelism / durable- replay / topic-introspection / programmatic-provisioning into load-bearing requirements. The migration was "straightforward because the transport abstraction had already decoupled our publishing and subscribing logic from the broker." Concrete framing of "scale changed the problem" applied to messaging primitive choice.
Last updated · 542 distilled / 1,571 read