Skip to content

ZALANDO 2026-10-07 Tier 2

Read original ↗

Thundering herd of circuit breakers

Summary

Zalando debugs a recurring production anomaly in which a client's circuit breaker stays open for hours, blocking ~95% of requests to a healthy downstream service that is handling barely any traffic (~5 RPS/pod). Two manual mitigations work — force-closing the breaker, or pre-scaling the service — but for reasons that aren't obvious from standard dashboards. By exporting raw OpenTelemetry trace data to CSV and building a bespoke single-page trace visualizer that plots hundreds of concurrent spans on an XY (time × sequential-trace) plot, the team sees the failure mode directly: the breaker's half-open probe phase sends only a trickle of requests, which a cold, minimally-scaled service handles fine; the breaker closes; the full withheld load arrives at once; the three under-provisioned pods queue and time out within ~3 seconds; the breaker re-opens. The root cause is a mismatch between the service the breaker measures (while probing) and the service that exists when the breaker closes (full load on minimum replicas) — a thundering herd created by the breaker's own close, which the CPU-averaging Horizontal Pod Autoscaler cannot react to because the overload burst is too short to move a minutes-averaged CPU metric. Framed as a variant of Google's SRE "low-traffic services" problem.

Key takeaways

  1. A circuit breaker's half-open probes validate the service as it is while probing, not as it will be when the breaker closes and full traffic returns. The probe trickle lands on a cold, minimum-replica service with plenty of headroom for a handful of requests — healthy spans (15 ms client, 10 ms reverse-proxy, 2 ms service) that meet the success threshold — but that measurement does not represent the service under the full load the close itself releases. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  2. The circuit breaker's own close is the synchronizer that creates a thundering herd. All traffic withheld while the breaker was open arrives in the same instant the breaker closes; the service goes "from receiving a handful of probes to being trampled by the thundering herd of heavy traffic released by the closed circuit breaker." This is a new TH shape on the wiki: the synchronizer is not a deploy, cache expiry, cron tick, or DNS TTL — it is the recovery mechanism's own state transition. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  3. CPU-averaging autoscaling can't see a 3-second overload burst. With ~5 RPS the HPA has scaled the service to its minimum of 3 pods. The full avalanche→timeout→re-open cycle "lasts about three seconds"; averaged over a minute, CPU "might register as just a small blip," so the autoscaler never scales out and the cycle repeats indefinitely. The named fix is to autoscale on queue depth or concurrency rather than CPU so the signal reacts in seconds, not minutes. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  4. Both manual mitigations work because both restore capacity before (or coincident with) full load. Force-closing the breaker floods real traffic back in, which the HPA now sees as sustained load and scales out; once enough warm pods exist, the breaker stays closed. Pre-scaling sizes the service for full traffic ahead of the breaker's next natural probe cycle, so the avalanche has capacity to land on. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  5. The service keeps processing requests the client already cancelled. When the client times out and drops a request, the service continues working on it — wasted work that makes queue growth worse during the avalanche. Named remediation: propagate cancellations so the service stops on client-abandoned requests. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  6. At low request rates, aggregate metrics stop being representative. The failure is explicitly framed as a variant of Google's SRE-workbook low-traffic services problem: at low volume, per-request variance dominates and percentiles are hard to interpret, so standard dashboards hid the recovery cycle entirely. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  7. High-fidelity, purpose-built visualization beats generic trace tooling for cross-request failure modes. Waterfall views show one trace at a time; aggregate percentiles smooth over the per-request detail. The team exported raw OTel CSV and vibe-coded an uPlot-based XY span plot (time on X, sequential traces on Y, timeouts red, opacity = duration) that made the probe→close→avalanche→re-open cycle visible as a single dense ~3-second burst. The broader lesson: AI agents using the same generic tooling only accelerate "hopeless grasping at straws," but agents used for guided bespoke tool development are a force multiplier. (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)
  8. A circuit breaker is not a self-contained solution — it makes environmental assumptions. It assumes probe traffic is representative of full traffic, that the protected service stays ready to receive load, and that recovery is gradual rather than instantaneous. When those assumptions break — as they do behind a scaled-to-minimum autoscaler — the breaker "can become the problem rather than the solution." (Source: sources/2026-10-07-zalando-thundering-herd-of-circuit-breakers)

Systems / concepts / patterns extracted

  • Circuit breaker — the central pattern; new failure mode: half-open probes don't reflect post-close capacity on an autoscaled service.
  • Thundering herd — the failure class; new shape where the synchronizer is the breaker's own close transition.
  • Elasticity / HPA autoscaling — CPU-averaged HPA at minimum replicas (3 pods) can't react to seconds-long bursts; autoscale on queue depth / concurrency instead.
  • Cold start — probes land on a cold, under-provisioned service; cold caches warm during the successful close.
  • Cascading failure — avalanche → queue growth → timeouts → breaker re-open → repeat is a positive-feedback loop driven by a protective mechanism.
  • Queueing theory — service processing stays 3 ms but requests queue >100 ms; cancelled-but-still-processed requests add wasted work that worsens queue growth.
  • Backpressure / load-shedding — named system-design fix: backpressure or load-shedding at the service boundary to keep queues from running away on a sudden burst.
  • Graceful degradation — the breaker is the resilience mechanism that, mis-tuned to its environment, degrades into the outage it was meant to prevent.
  • Observability — bespoke OTel-CSV trace visualizer; the limits of waterfall-per-trace tooling and aggregate percentiles for cross-request failure analysis.
  • OpenTelemetry — raw trace data exported to CSV (via Dash0) as the substrate for the bespoke visualizer.
  • Kubernetes — HPA + minimum replica count is the autoscaling substrate under the service.

Operational numbers

  • Request rate during the anomaly: ~5 RPS per pod; cluster baseline load "thousands of req/s" elsewhere.
  • Breaker held ~95% of requests back while open; open "for hours."
  • HPA minimum: 3 pods; each pod then forced to absorb ~10× its probe-era load on close.
  • Half-open probe span latencies (all healthy): 15 ms client, 10 ms reverse-proxy, 2 ms service.
  • Avalanche queuing: service processing latency ~3 ms, but requests queue >100 ms before processing.
  • Full probe→close→avalanche→re-open cycle duration: ~3 seconds — short enough to vanish in a minute-averaged CPU metric.
  • Trace window analyzed: a 10-minute window around the final successful close (three failed close attempts + one successful).

Caveats

  • This is a debugging / architecture narrative, not a benchmark: no post-fix results are reported. The system-design remedies (queue-depth autoscaling, longer/heavier half-open probe windows, pre-scaling, backpressure/load-shedding, cancellation propagation) are proposed, not yet measured in the post.
  • Numbers (5 RPS/pod, 3 pods, 3-second cycle, span latencies) are single-incident observations from one service pair, not a controlled study.
  • The specific circuit-breaker library / HPA configuration (thresholds, probe volume, cooldown) are not disclosed, so the exact parameter values that make the trap fire are illustrative of the shape rather than a reproducible recipe.
  • The bespoke trace visualizer is open-sourced as an example but consumes Dash0 CSV exports specifically.

Source

Last updated · 854 distilled / 2,448 read