Skip to content

CONCEPT Cited by 1 source

Streaming-first architecture

Streaming-first architecture is the design decision to build a system's primary data path as a continuous streaming pipeline rather than periodic batch jobs, even when batch would be simpler. The key trade-off: increased implementation complexity in exchange for data freshness (tens of minutes vs. hours).

When streaming-first is load-bearing

Netflix argues streaming-first is essential — not optional — for observability systems used during incidents:

  • "During a production incident at 3am, an hour-old dependency map is archaeology, not observability."
  • Live events can't wait for the next hourly batch.
  • Change validation requires seeing immediate impact.
  • Incident response needs current data.

(Source: sources/2026-07-13-netflix-building-service-topology-at-scale-architecture-challenges)

Backpressure enables streaming-first at scale

The fundamental challenge: how do you process millions of records/sec in real-time without losing data when downstream systems slow down?

Streaming-first + backpressure provides the answer: when any stage is overwhelmed, the pipeline automatically slows to a sustainable rate. Data waits in Kafka until capacity returns. No data loss, no crashes — slightly delayed updates instead of dropped records.

"During normal operation, we process with minimal latency. During load spikes or temporary slowdowns, we slow down rather than fall over."

Trade-off vs. batch

Batch Streaming-first
Freshness Hours Minutes
Complexity Lower Higher (backpressure, reactive streams)
Failure mode Missing batch = stale data Slow pipeline = slightly delayed data
During incidents Archaeology Actionable
Operational cost Simpler to reason about Requires deep streaming expertise

Seen in

Last updated · 585 distilled / 1,765 read