Skip to content

CONCEPT Cited by 1 source

In-memory coalescing by Kafka key

In-memory coalescing by Kafka key is the technique of merging multiple Kafka messages that share the same key into a single aggregate inside a consumer's batch buffer before flushing to downstream storage. This reduces downstream write volume and amortises transactional / I/O overhead over multiple input messages.

The mechanism is only safe when two conditions hold:

  1. Same-key messages land on the same partition. Kafka guarantees hash-of-key-to-partition-per-topic, so any consumer reading a partition sees all messages for a given key in-order and in-batch.
  2. The aggregate operation is mergeable. For counts, sums, and DDSketches this is natural (monoid-like merge); for percentiles on raw samples it is not.

(Source: sources/2026-04-21-planetscale-storing-time-series-data-in-sharded-mysql-to-power-query-insights.)

PlanetScale's canonical application

Rafer Hazen, 2023-08-10: "Aggregate query data is mapped to Kafka partitions by setting the Kafka key to a deterministic hash of the database identifier and the query fingerprint. Because of this, all messages for a given database/query pattern will arrive in the same partition and we can merge aggregate Kafka messages in memory for each consumer batch to avoid unnecessary database writes. In practice, we've found that in-memory coalescing decreases database writes by about 30%–40%."

The same key is hash(database_id, query_fingerprint) — so every (database × pattern) gets its own natural grouping lane. A single consumer batch of ~200 Kafka messages representing say 150 distinct patterns writes ~150 aggregated rows to MySQL, not 200.

Batch-size vs coalesce-rate trade-off

Larger consumer batches coalesce more:

  • Average batch ~200 messages, typical operation.
  • Up to 1,000 messages per batch during load spikes or backlog burn-down.

Hazen canonicalises the self-reinforcing dynamic: "Larger batches yield better write coalescing but require more memory in the consumer and increase end-to-end latency … The higher coalesce rate in larger batches helps us quickly burn down message backlogs when they occur." The system naturally sheds its own write-amplification under load — this is the Kafka consumer equivalent of a load-shedding circuit.

Operational effect — 30–40% DB-write reduction

PlanetScale's published number: "in-memory coalescing decreases database writes by about 30%–40%." For a Kafka consumer that sends ~5k writes/s to MySQL at steady state, that's ~2–3k writes/s of unnecessary work avoided — the same load-bearing economic figure that keeps a 2 vCPU / 2 GB per-shard fleet viable.

Composition with idempotence

Coalescing is orthogonal to the idempotence guarantee (Kafka offset + partition uniqueness — see concepts/kafka-offset-partition-uniqueness-constraint) — coalescing runs in-memory on the consumer, idempotence guards the final DB write. If the DB write fails after a coalesce and the batch retries, the uniqueness constraint dedup applies to the coalesced row, not the pre-coalesce messages.

Seen in

Last updated · 470 distilled / 1,213 read