Skip to content

PATTERN Cited by 2 sources

Single-trigger latency-class switch

Definition

A single-trigger latency-class switch is an architectural property of a stream-processing framework where the latency class of a pipeline (second-range micro-batch vs millisecond-range continuous) is controlled by a single configuration parameter (the trigger type), with no code rewrite, no checkpoint format change, and no new API surface required.

The pattern decouples the what (streaming logic, state management, source/sink integration) from the how fast (execution scheduling model), letting teams prototype at micro-batch latency and promote to sub-millisecond by changing one line.

Shape

# Second-range (micro-batch) — existing code
query.start(trigger="5 seconds")

# Millisecond-range (RTM) — same code, one config change
query.start(trigger="realtime", interval="5 minutes")

Same Structured Streaming API. Same checkpoint management. Same delivery semantics. Only the trigger configuration differs.

When to use

  • When an existing micro-batch pipeline needs to be promoted to operational latency without migration risk
  • When the same pipeline code should serve both analytical (cost-optimised) and operational (latency-optimised) deployments
  • When teams want to evaluate latency improvement before committing to infrastructure changes

Trade-offs

  • RTM trigger imposes constraints: fixed worker count (no autoscaling), dedicated cluster, Photon disabled, update output mode required.
  • At-least-once delivery (vs micro-batch's exactly-once option with some sinks).
  • Cost-effectiveness: micro-batch is cheaper for workloads tolerating 1–2 seconds; RTM is justified only when sub-second latency has direct business impact.

Known uses

Last updated · 585 distilled / 1,765 read