PATTERN Cited by 1 source

Fast-lane / slow-lane consumer split

Definition

Split a single Kafka consumer group into two parallel consumer groups reading from the same topic: a fast lane that skips messages identified as slow-to-process, and a slow lane that handles only those slow messages. This eliminates head-of-line blocking without adding Kafka partitions.

Mechanism

1. Both consumer groups subscribe to the same topic/partitions.
2. On message receipt, the fast-lane consumer performs a cheap classification (e.g., checking message metadata or a lightweight lookup) to determine if the message will be slow.
3. If slow → the fast-lane consumer skips it (commits the offset without processing); the slow-lane consumer will process it.
4. If fast → the fast-lane consumer processes it normally; the slow-lane consumer skips it.

Trade-offs

Pro	Con
No Kafka partition increase needed	Two consumer groups → 2× partition assignments
Fast messages never blocked by slow ones	Classification must be cheap and accurate
Each lane can be scaled independently	Offset management is more complex
Simple to implement	Slightly higher broker load (two groups reading same data)

When to use

• A topic mixes messages with bimodal processing-time distributions (e.g., accounts with 10 assets vs. 500,000 assets).
• Adding partitions is undesirable (shared broker, resource constraints).
• The fast/slow determination can be made from message headers or a quick lookup, not from full processing.

Seen in

• sources/2026-06-12-cloudflare-scaling-security-insights — Cloudflare's Security Insights checkers split into fast-lane and slow-lane consumer groups. One account may have far more assets than another; the slow lane processes those without blocking the fast lane.