Skip to content

NETFLIX 2026-07-13

Read original ↗

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

Summary

This is the second post in Netflix's Service Topology series — a deep engineering deep-dive into how the system was built at production scale. Where the first post described what (three-layer graph: eBPF network flows, IPC metrics, distributed tracing) and why (troubleshoot faster, understand blast radius), this post details the streaming-first architecture, the three-stage distributed aggregation pipeline, and the V1/V2 production challenges encountered scaling the system (Kafka consumer lag, hot nodes from power-law traffic distribution, garbage-collection pressure, reactive-streams complexity). It also covers the time-travel feature enabling point-in-time topology reconstruction.

Key takeaways

  1. Streaming-first beats batch for observability. Netflix chose real-time streaming over hourly batch to deliver topology freshness within tens of minutes, not hours. During incidents, hour-old data is "archaeology, not observability." Backpressure allows graceful degradation without data loss when load spikes hit. (§ Streaming-First)

  2. Three-stage aggregation pipeline solves both intermediary resolution AND hot nodes. The middle stage (Stage 2) resolves network intermediaries (LBs, NAT gateways, proxies) by grouping and joining inbound/outbound flows. But the architecture also distributes load across three redistribution boundaries — each stage re-hashes and re-distributes, preventing data concentration on any single instance even with power-law traffic. (§ Three-Stage Pipeline, § Challenge 2)

  3. Server-Sent Events replaced gRPC for inter-stage communication. gRPC's serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. SSE — lightweight HTTP-based protocol — proved ideal for streaming pre-aggregated data with natural backpressure integration. "Measure, don't assume" — industry best practices don't apply universally. (§ Why SSE Instead of gRPC)

  4. Dynamic consistent hashing via service registry eliminates coordination. Instead of static cluster membership or Zookeeper-like coordination, instances query the service registry for current ASG membership, sort the list identically, and hash into it. ASG scale-out/in automatically redistributes aggregators. (§ Dynamic Load Distribution)

  5. Mutable data structures on the hotpath cut GC pressure >50%. Scala/JVM best-practice immutability creates too many short-lived objects at millions of records/second. Netflix pragmatically switched to mutable aggregators on the hot path (immutability elsewhere), reducing heap allocation by over 50% and dramatically cutting GC pause times. Done deliberately with measurement, not by default. (§ Challenge 3)

  6. Cascading bottlenecks are the nature of distributed systems at scale. Fixing Kafka lag exposed hot nodes. Fixing hot nodes exposed GC. Fixing GC exposed serialization issues. Each optimization raises throughput, stressing the next weakest link. The approach: measure, fix one bottleneck thoroughly, move to the next. (§ Lessons)

  7. Time-travel queries via windowed snapshots + property-level mutations. Immutable 5-minute aggregator snapshots keyed by (entity_id, timestamp) combined with sparse property-level mutation history enable arbitrary historical topology reconstruction without full-snapshot storage costs or slow log replay. Query-time re-aggregation reuses ingestion aggregator classes for flexible groupby dimensions. (§ Time Travel)

  8. Data partitioning strategy determines processing architecture. IPC metrics arrive pre-partitioned at application level → single-stage aggregation works. Network flows arrive partitioned by Kafka key (not by intermediary) → require shuffle/redistribution stages to bring related flows together. (§ Why IPC Doesn't Need Three Stages)

  9. Reactive streams are powerful but require deep investment. Non-intuitive demand-driven model, async boundary complexity, and silent stalls made Pekko Streams hard to debug. Netflix invested in team education, simplified stream graphs, and added internal stream metrics. Worth it for systems that need graceful load handling. (§ Challenge 4)

  10. Scale changes everything qualitatively. Immutable data structures that are fine at 100 rps create GC thrashing at 100K rps. Single-stage aggregation that works in dev fails catastrophically with power-law production traffic. Standard gRPC becomes heavyweight for streaming aggregation at volume. Break conventions only when measurements justify it. (§ Lessons)

Operational numbers

  • Processes millions of flow records per second from multi-region Kafka
  • 4 Kafka regions feeding the flow-log pipeline
  • 5-minute time windows for aggregation batching
  • Popular services generate 100× more traffic than typical services (power-law)
  • Sub-second query response times
  • Near real-time freshness (tens of minutes vs. hours)
  • GC optimizations: mutable aggregators reduced heap allocation >50%, cut pause times from hundreds of ms to tens of ms

Architecture extracted

Three-stage distributed aggregation pipeline

Stage Role Key operation Redistribution
Stage 1 (FlowLog Ingestion) Initial aggregation from Kafka 5-min time-window batching, filter invalid logs Consistent hash → Stage 2 via SSE
Stage 2 (Intermediate GraphEntity) Network intermediary resolution Group flows by intermediary, join inbound+outbound → direct edges Re-hash → Stage 3 via SSE
Stage 3 (GraphEntity Ingestion) Final aggregation + enrichment Enrich from external KV stores, convert to graph entities Throttled writes → graph DB

IPC pipeline (single-stage)

IPC metrics are already at application level — no intermediary resolution needed. Data arrives correctly partitioned from the start. Single-stage aggregation → enrichment → persist.

Time-travel architecture

  1. Time-windowed aggregator snapshots — immutable, keyed by (entity_id, timestamp)
  2. Property-level mutation tracking — sparse, stores only changed properties with timestamps
  3. Query-time reconstruction — fetch mutations for time range, apply in order

Caveats

  • This post covers the network layer (eBPF flows) and IPC layer. The tracing layer integration is deferred to a third post.
  • Specific numeric improvements (e.g., exact throughput gains from SSE vs gRPC) are described qualitatively rather than with precise benchmarks.
  • The mutable-hotpath trade-off requires "more careful code review" — it's not a blanket recommendation.

Source

Last updated · 585 distilled / 1,765 read