SYSTEM Cited by 18 sources

Apache Kafka

Distributed, partitioned, replicated append-only log; the canonical open-source streaming-messaging substrate. Producers write keyed records to topics, which are split into partitions for horizontal scale; consumers read from partitions with at-least-once (default) or exactly-once semantics. Keyed records with the same key land on the same partition (hash-of-key ⇒ partition index), which is the foundation that higher layers (like systems/kafka-streams) build partition-local guarantees on top of.

Core primitives (referenced by Kafka Streams analyses)

• Topic — named, append-only logical log.
• Partition — unit of parallelism; each topic is split into N partitions; ordering is guaranteed within a partition only.
• Record key — the hash of the key determines the partition. Identical keys land on the same partition of a single topic by construction.
• Consumer group — set of consumers that cooperatively divide partitions of a topic; Kafka Streams is layered on top of this primitive.

Cross-topic keying (the Expedia sub-topology lesson)

Two topics with identical partition counts and similarly-keyed records do not by themselves guarantee that the same key lands on the same consumer instance across the two topics — that is a property of the consuming framework, not of Kafka itself. Kafka only guarantees same-key-to-same-partition per topic. When systems/kafka-streams is the consumer, the extra colocation guarantee is sub-topology-scoped — see concepts/partition-colocation and sources/2025-11-11-expedia-kafka-streams-sub-topology-partition-colocation.

Batching semantics — what Kafka does and doesn't do

Kafka's producer-side batching is byte-count + message-count + time-window within a partition:

• batch.size — max bytes buffered per partition before dispatch.
• linger.ms — max wait to accumulate a batch.
• max.in.flight.requests.per.connection — pipeline depth.

These compose to a transport-economics batching primitive (saturate TCP, amortise broker bookkeeping) but don't express payload-attribute budgets — "Kafka batches by bytes/messages within a partition; token count varies with text and tokenizer, so there is no efficient way to batch requests by Σ token_count_i" (2025-12-18 Voyage AI). For application-specific batching such as token-count batching for GPU inference, the pattern is to keep Kafka for durability / fan- out / delivery and insert a lightweight aggregator between Kafka and the workers that runs application batching logic.

Seen in

• sources/2026-06-02-redpanda-how-omninode-uses-redpanda-to-scale-ai-agent-workflows — 2026-06-02 OmniNode founder guest post on Redpanda Blog. OmniNode adopts the Kafka API via Redpanda after outgrowing Redis Streams at 100+ event types / 12 repos. Canonical wiki disclosure that the Kafka-API set of capabilities — consumer groups, partition-level parallelism, durable replay semantics, topic introspection, programmatic provisioning — is what makes cross-repo / multi-agent coordination possible at scale. Crucial caveat: "the topic name is the only thing connecting one agent's output to another agent's input"; the canonical wiki disclosure of topic name as coordination surface and the silent wiring failure bug class (routing-complete vs routing_complete accepted by the broker as different topics, both sides green, no events flow). OmniNode's response — contract-driven topic provisioning with regex + StrEnum validation and one parser, three call sites (CI / runtime boot / post-boot validation) — is a generic discipline applicable to any Kafka-API broker. Pattern is broker-agnostic; the Redpanda angle in the post is the single-binary affordability that lets the broker exist in dev / CI / prod identically: "if the broker is operationally heavy, teams eventually stop running it locally."

• sources/2025-04-23-redpanda-need-for-speed-9-tips-to-supercharge-redpanda — 2025-04-23 Redpanda omnibus performance-tuning checklist. Kafka-API-compatible (every tip applies identically to Kafka). Canonicalises partition skew as the parallelism-killer; the four-parameter consumer fetch-tuning matrix; the save-button / read-becomes-write analogy for __consumer_offsets commits; ZSTD / LZ4 codec guidance; the compression + compaction CPU interaction; and tiered storage as decommission / recommission accelerator. Introduces the concepts/keyed-partitioner concept page and patterns/client-side-compression-over-broker-compression pattern page.

• sources/2025-02-11-redpanda-high-availability-deployment-multi-region-stretch-clusters — canonical wiki instance of the Kafka-API's multi-region high-availability shape via per-partition Raft quorum across regions. Redpanda ships the Kafka wire protocol; the four stretch-cluster operator knobs this post canonicalises all have upstream-Kafka equivalents or extensions: acks=1 is the Kafka producer- side durability dial; follower fetching is KIP-392 (Kafka 2.4+, client.rack-driven closest-replica consume); MirrorMaker2 is Apache Kafka's cross-cluster async replication tool ("multiple independent Redpanda clusters across different regions with MM2 replication"); leader pinning and remote read replica topic are Redpanda productisations without direct single-knob Kafka equivalents (upstream Kafka approximates via preferred-replica-election + rack-aware placement). Canonicalises the stretch-cluster-vs- MirrorMaker2 axis as the first-order consistency-vs-availability architectural choice on a Kafka-API cluster spanning regions.

• sources/2024-11-19-redpanda-batch-tuning-in-redpanda-for-optimized-performance-part-1 — canonical wiki source for Kafka-API producer-side batching substrate. James Kinley (Redpanda, 2024-11-19) frames batching from first-principles request economics (fixed + variable cost), walks the linger.ms / batch.size / buffer.memory trigger logic as pseudo-code, and canonicalises the seven-factor framework for effective batch size in production. The Redpanda framing is Kafka-API-compatible; every claim applies identically to Apache Kafka producers. Also canonicalises the counterintuitive latency inversion under CPU saturation — increasing linger.ms can reduce tail latency when the broker is saturated by shrinking the internal work-queue backlog — and backpressure-driven batch growth (saturated broker → in-flight slots occupied → producer queues records past batch.size).

• — Canonical wiki instance as the message bus in a dual-stream database-telemetry pipeline. PlanetScale Insights (a Postgres extension) emits to two separate Kafka topics: (1) individual queries — one message per notable-tail event (query read > 10k rows OR > 1s OR error), and (2) aggregate summaries — one message per query pattern per unique tag combination every 15s (post-this-release; before this release it was one message per query pattern). Both topics feed ClickHouse. Two separate topics rather than one with event-type routing reflects the different volume / retention / schema-evolution profiles — the pattern explicitly.

• sources/2025-12-18-mongodb-token-count-based-batching-faster-cheaper-embedding-inference — named alongside RabbitMQ as a general-purpose broker whose native batching primitives don't fit token-count batching of GPU embedding-inference requests. Voyage AI chose a native store (Redis + Lua, see patterns/atomic-conditional-batch-claim) rather than put an aggregator in front of Kafka — but either path works, and the aggregator variant keeps Kafka's durability.
• sources/2025-11-11-expedia-kafka-streams-sub-topology-partition-colocation — Expedia's production debugging case where "same partition count + similar keying across two topics" was assumed (incorrectly) to imply cross-topic colocation at the consumer-instance level; the missing ingredient turned out to be a Kafka-Streams-layer constraint (shared sub-topology), not a Kafka-broker one.
• sources/2025-11-04-datadog-replication-redefined-multi-tenant-cdc-platform — Kafka as the durable transport middle of Datadog's managed multi-tenant CDC replication platform. Debezium source connectors on Kafka Connect publish Avro-serialised record streams to Kafka topics (validated against a Kafka Schema Registry in backward-compat mode); sink connectors drain topics into Elasticsearch / Postgres / Iceberg / Cassandra / cross-region Kafka. Canonical wiki instance of patterns/debezium-kafka-connect-cdc-pipeline. Cross-region Kafka replication also cited as an in-platform sink for Datadog On-Call data locality + resilience.
• sources/2026-01-06-lyft-feature-store-architecture-optimization-and-evolution — Kafka (alternatively Kinesis) as the event-source upstream of streaming feature ingestion in Lyft's Feature Store. Customer Flink applications read analytic events from Kafka topics, transform them, and emit feature payloads into Lyft's central spfeaturesingest Flink app, which writes to dsfeatures via a WRITE API. Canonical Kafka-as-feature-store-streaming-source instance.
• sources/2025-04-08-netflix-how-netflix-accurately-attributes-ebpf-flow-logs — Kafka as a cluster-broadcast bus for eventually-consistent shared state in Netflix's eBPF flow-log attribution rebuild (patterns/kafka-broadcast-for-shared-state). Every FlowCollector node publishes learned (ip, workload_id, t_start, t_end) time ranges to a broadcast topic; every node consumes the topic and merges peer time ranges into its local per-IP ownership map. Netflix's explicit acknowledgement that "more efficient broadcasting implementations exist" but Kafka "is simple and has worked well for us" canonicalises the simplicity-over-optimality trade-off.
• sources/2026-04-04-netflix-powering-multimodal-intelligence-for-video-search — Kafka as the offline-fusion trigger bus inside Netflix's multimodal video-search pipeline. On every raw-annotation write by Marken, an event is published via Kafka to trigger an asynchronous fusion job that discretizes per-model annotations into fixed-size time buckets and computes cross-model intersections. A second Kafka event triggers indexing of enriched buckets into Elasticsearch. Canonical wiki instance of patterns/offline-fusion-via-event-bus — Kafka's role is decoupling heavy intersection computation from ingest such that "complex data intersections never bottleneck real-time intake." Topic topology, partition count, retention, and ordering guarantees not disclosed in the 2026-04-04 post.

Origin + substrate (2024-05-09 Kozlovski Kafka-101)

Stanislav Kozlovski's Kafka-101 explainer on High Scalability (2024-05-09) is the wiki's canonical architectural tour of Kafka. Key framings this system inherits:

• Origin: "Originally developed in LinkedIn during 2011, Apache Kafka is one of the most popular open-source Apache projects out there. So far it has had a total of 24 notable releases and most intriguingly, its code base has grown at an average rate of 24% throughout each of those releases." Designed to solve the service-coordination problem as a single platform which can serve as the source of truth / central nervous system inside a company, optimised for "large throughput (millions of messages a second) while storing a lot of data (terabytes)."
• Log as data model — a topic is a log (ordered, append-only, immutable, O(1) at head/tail). Log chosen because it's optimised for HDDs and decouples producer and consumer lifetimes.
• Broker / partition / replica — a broker hosts partitions; each partition has R replicas; one replica is leader (leader-follower replication), the rest are followers; the in-sync replica set (ISR) is the subset caught up with the leader.
• Producer durability dial — acks=0/1/all composable with min.insync.replicas for fail-closed durability against ISR shrink.
• Consumer groups — consumer-group primitive with partition-exclusivity lifts per-partition ordering into end-to-end ordering; offsets persisted to __consumer_offsets.
• Controller + consensus — one broker is the active Controller owning leader-election. Historical ZooKeeper backend (systems/apache-zookeeper) retiring in favour of in-tree KRaft with cluster metadata as a Kafka log (production-ready Kafka 3.3, Oct 2022; full removal in Kafka 4.0).
• Performance stack — protocol batching (patterns/batch-over-network-to-broker) + OS read-ahead + write-behind + pagecache + sendfile zero-copy (the last one disabled by production TLS — Kozlovski's honest-assessment caveat).
• Tiered Storage — Early-Access feature pulling historical segments into object storage (e.g. S3) to break the four structural walls of co-located broker storage (log recovery, historical-read IOPS exhaustion, full re-replication on disk failure, rebalance motion). See patterns/tiered-storage-to-object-store + concepts/log-recovery-time. 43% producer performance improvement when historical consumers were present, per development benchmarks.
• Rebalancing is NP-hard → Cruise Control (LinkedIn-originated open-source bin-packer, prioritised Goals against per-broker metrics).
• Framework ecosystem — Kafka Connect (source/sink connectors)
• Kafka Streams (JVM library, exactly-once when input + output are Kafka topics).

• Industry trajectory — "evidence points that the future will be everybody standardizing on the Kafka API and competing on the underlying implementation." Named alternatives: Kora (Confluent cloud-native), systems/redpanda (C++ rewrite), WarpStream (S3-heavy, stateless brokers).

• sources/2026-04-23-aws-modernizing-kyc-with-aws-serverless-solutions-and-agentic-ai — IBM + AWS KYC architecture uses Amazon MSK as the async communication backbone for a Supervisor + five KYC sub-agents. Canonical AWS instance of the agentic-AI coordination shape — four inbound topic categories (KYC requests, document uploads, ID-verification results, transaction events) + three outbound categories (decisions, escalations, fraud alerts); event-listener pre-processing; Lambda consumers that invoke AgentCore asynchronously. See patterns/inbound-outbound-topic-pairing, patterns/async-agent-invocation-over-kafka, patterns/multi-agent-streaming-coordination.

• sources/2026-05-04-netflix-democratizing-machine-learning-building-the-model-lifecycle-graph — Kafka as the change-notification substrate for Netflix MDS. Six source systems (Pipeline Orchestration, Model Registry, Feature Store, Experimentation Platform, Datasets, Identity Platform) emit thin notification-of-change events (just {event_type, instance_id}) via Kafka + SNS / SQS; MDS hydrates full state from source APIs on receipt. Reinforces Kafka's role as the canonical event-bus substrate for thin-event + source-hydration architectures where the event log is a trigger, not a state log.
• sources/2026-05-29-netflix-from-silos-to-service-topology-why-netflix-built-a-real-time-service-map — multi-region Kafka as the eBPF flow-log ingestion substrate for Netflix Service Topology. "We consume flow logs from Kafka across multiple AWS regions where Netflix operates. This runs continuously, processing millions of flow records as they arrive." Service Topology's three-stage Pekko Streams aggregation pipeline reads from this Kafka substrate as Stage 1's input. Sibling Kafka use case to the 2025-04-08 FlowCollector pipeline, which uses Kafka broadcast for shared-state (patterns/kafka-broadcast-for-shared-state) — together the two posts canonicalise Kafka's two distinct roles in Netflix's flow-log substrate: ingestion-pipe (Service Topology) and cross-node-state-broadcast (FlowCollector).

• sources/2026-05-29-netflix-high-throughput-graph-abstraction-at-netflix-part-i — Kafka as an entropy-repair substrate for Netflix Graph Abstraction. A graph edge write touches 3+ KV records across separate namespaces (forward link / reverse link / edge property); without distributed transactions, partial-write failures need durable retry-until-convergence. Verbatim: "To prevent inconsistencies or lasting entropy from failures in any operation, the Abstraction uses a robust retry mechanism using Kafka." This canonicalises a third Kafka role — durable retry queue for cross-namespace consistency — distinct from event-log, stream-processing, and CDC framings elsewhere on the wiki. Composes with idempotency tokens + LWW to deliver strict eventual consistency across multi-namespace graph writes (patterns/kafka-entropy-repair-for-multi-namespace-writes).

• sources/2026-06-03-databricks-apache-spark-real-time-mode-for-gaming — Kafka serves as both input (gaming session events from consoles/PCs) and output (processed SessionActive / SessionEnd events) for a Spark Real-Time Mode sessionization pipeline running at 500K input events/min with 432 ms p99 end-to-end Kafka-to-Kafka latency.

Log compaction correctness bug (Kafka 3.9–4.2)

A compaction–replication race disclosed by Redpanda (2026-06-25) demonstrates that Kafka's per-broker, uncoordinated log compaction can cause permanent replica divergence on compacted topics with transactional writes. If a broker is offline longer than delete.retention.ms (default 24 h), tombstones or transaction control batches may be compacted away before the broker replicates them, producing four distinct failure modes: deleted data reappears, aborted data served as committed, committed data hidden, or read_committed consumers frozen at a stale Last Stable Offset. The bug reproduces reliably; no parameter-tuning fix exists because no finite retention value guards against unbounded unavailability. (Source: sources/2026-06-25-redpanda-kafkas-log-compaction-corrupts-data)

Related

• systems/kafka-streams — Kafka-native stream-processing framework
• systems/kafka-connect — Kafka-native connector framework (hosts Debezium + sink connectors)
• systems/debezium — Kafka Connect-based CDC source connector family
• systems/kafka-schema-registry — Avro-schema gating integrated with producers + consumers
• systems/apache-zookeeper — retired coordinator
• systems/kraft — in-tree Raft-dialect consensus replacing ZooKeeper
• systems/cruise-control — LinkedIn's open-source cluster rebalancer
• systems/warpstream / systems/confluent-kora / systems/redpanda — alternative Kafka-API implementations
• concepts/partition-colocation
• concepts/sub-topology
• concepts/change-data-capture
• concepts/distributed-log / concepts/kafka-partition / concepts/in-sync-replica-set / concepts/consumer-group / concepts/leader-follower-replication / concepts/acks-producer-durability / concepts/kraft-metadata-log / concepts/pagecache-for-messaging / concepts/log-recovery-time / concepts/hdd-sequential-io-optimization
• systems/rabbitmq — sibling general-purpose broker with push-model prefetch batching
• patterns/lightweight-aggregator-in-front-of-broker — canonical shape for application batching (e.g. token-count batching) on top of Kafka
• patterns/debezium-kafka-connect-cdc-pipeline — canonical CDC pipeline shape built on Kafka
• patterns/append-only-log-as-substrate / patterns/leader-based-partition-replication / patterns/tiered-storage-to-object-store / patterns/consumer-group-partition-exclusivity / patterns/batch-over-network-to-broker / patterns/zero-copy-sendfile-broker — architectural patterns Kafka instantiates
• concepts/token-count-based-batching
• concepts/multi-region-stretch-cluster / concepts/leader-pinning / concepts/follower-fetching / concepts/remote-read-replica-topic / concepts/mirrormaker2-async-replication / concepts/cross-region-bandwidth-cost — multi-region HA/DR shape and operator knobs canonicalised on the Kafka-API via Redpanda's 2025-02-11 stretch-clusters post
• patterns/multi-region-raft-quorum / patterns/client-proximal-leader-pinning / patterns/closest-replica-consume / patterns/tc-latency-injection-for-geo-simulation — patterns for Kafka-API multi-region deployments
• systems/openmessaging-benchmark — driver-abstracted benchmark framework for Kafka and Kafka-API-compatible brokers
• systems/zerobus-ingest — Databricks' serverless streaming alternative that explicitly bypasses Kafka by replacing static partition-level ordering with stream-connection-level ordering, enabling true elastic autoscaling (Source: sources/2026-06-11-databricks-ingesting-the-milky-way-petabyte-scale-with-zerobus-ingest)

Tiered Storage at 150B events/day (Atlassian StreamHub, 2026)

Atlassian's StreamHub runs Kafka via MSK at 150 billion events/day (3.2M events/sec peak). Their Tiered Storage deployment exposed critical operational lessons:

• Local tier = 5 minutes on EBS (hot, real-time consumers); Remote tier = S3 with 7-day retention (historical reads, backfills).
• Tiered Storage is a cost and retention strategy, not a capacity strategy: if brokers are saturated, the async remote-copy workers fall behind, causing linear local disk growth.
• S3 delete storms: reducing retention triggers mass deletes that compete with regular remote writes, stalling offload and filling local disks. Treat retention changes as production risk operations.
• Broker headroom is a reliability feature: replication, consumer fan-out, cross-region relay, retries, and tiered storage offload all compete for the same broker network/EBS budget. Plan around peak per-broker metrics, not cluster averages.

Key operational patterns: patterns/conservative-broker-headroom, patterns/failover-cluster-escape-hatch, patterns/cluster-sharding-for-blast-radius, patterns/ingress-rate-limiting-and-quarantine.

(Source: sources/2026-07-28-atlassian-scaling-streamhub-transitioning-from-kinesis-to-kafka)