PATTERN Cited by 1 source

Batch-goroutine parallel consumption

Definition

Consume Kafka messages in batches rather than one-at-a-time, and process each message in the batch concurrently via goroutines (or equivalent lightweight threads). This extracts parallelism from a single partition consumer without violating Kafka's per-partition ordering guarantee at the consumption level.

Mechanism

1. Consumer fetches N messages from the partition in one poll.
2. Each message is dispatched to a separate goroutine for processing.
3. The consumer waits for the entire batch to complete before committing offsets.
4. Offset commit covers the highest offset in the batch.

Trade-offs

Pro	Con
N× throughput per partition consumer	If process crashes mid-batch, all N messages are re-processed
No additional Kafka partitions needed	Higher memory usage (N concurrent processing contexts)
Simple implementation in Go	Processing order within batch is non-deterministic
Works with existing partition count	Batch size must be tuned to avoid OOM

When to use

• Per-message processing is I/O-bound (API calls, DB writes) rather than CPU-bound.
• Strict ordering between messages in the same batch is not required (only at-least-once semantics needed).
• The consumer language has cheap concurrency primitives (Go goroutines, Kotlin coroutines, etc.).

Seen in

• sources/2026-06-12-cloudflare-scaling-security-insights — Cloudflare's Go-based security checkers consume Kafka messages in batches, processing each in a separate goroutine. Trade-off accepted: more re-work on crash, slightly higher memory.