SYSTEM Cited by 1 source

WarpStream

WarpStream is a Kafka-API-compatible streaming system that takes tiered storage to its logical extreme: all data lives directly in an object store (e.g. S3), brokers are stateless, and there is no broker-to-broker replication — durability is inherited from the object store.

Stub page — expand on future WarpStream-architecture sources. Canonical wiki entry point is Kozlovski's 2024-05-09 Kafka-101 explainer.

Design endpoint

Kozlovski, 2024-05-09:

"WarpStream, which innovated with a new architecture that leverages S3 heavily, completely avoiding replication and broker statefulness." (Source: sources/2024-05-09-highscalability-kafka-101)

This is the far end of the Tiered-Storage design axis — where the Kafka-101 post's Tiered Storage section pulls historical segments into S3 while keeping the hot tail on local disk, WarpStream pulls everything into S3 and treats brokers as stateless compute. The shape is explicitly stateless-compute at the broker tier.

Seen in

• sources/2024-05-09-highscalability-kafka-101 — named among the three canonical alternative implementations of the Kafka API. Positioned as the most-divergent-from-classical-Kafka architecture — the Tiered-Storage-extreme design point.

Related

• systems/kafka — upstream API WarpStream implements.
• systems/aws-s3 — canonical object-store substrate.
• systems/confluent-kora / systems/redpanda — sibling alternative implementations of the Kafka API at different points on the implementation-divergence axis.
• patterns/tiered-storage-to-object-store — the pattern WarpStream takes to an extreme.
• concepts/stateless-compute — the broker-tier consequence of pushing all state into object storage.