Skip to content

AWS 2026-07-15

Read original ↗

How bitdrift scaled to 121 million concurrent gRPC connections on Amazon CloudFront

Summary

bitdrift, a mobile observability platform founded by former Lyft infrastructure engineers, hit a critical scaling failure while serving real-time telemetry during the T20 World Cup cricket series. Their architecture — CloudFront at the edge, multiple NLBs at the origin, gRPC for persistent connections from mobile devices — was sound, but Route 53's Weighted routing policy returned only a single IP per DNS response. This caused all CloudFront edge nodes to resolve the origin to the same NLB for the duration of the DNS TTL, creating a thundering herd that overwhelmed individual load balancers. The fix was purely configurational: switching to Multi-Value Answer routing, which returns up to 8 IPs per response with built-in health checks, spreading origin connections across all NLBs from the first DNS resolution. After the change, bitdrift handled 121 million concurrent devices with zero server-side errors.

Key takeaways

  1. DNS routing policy selection is a load-bearing architectural decision at extreme scale. Weighted routing (1 IP/response) vs Multi-Value Answer routing (up to 8 IPs/response) has dramatically different origin load distribution behavior behind a CDN like CloudFront. At normal scale this is invisible; at 121M devices it's the difference between zero errors and 80% failure rate. (Source: comparison table and before/after metrics in original.)

  2. Persistent connections (gRPC, WebSocket) amplify DNS routing problems catastrophically. Unlike short-lived HTTP requests where connections churn and naturally redistribute, gRPC connections are long-lived and accumulate on whichever origin was resolved at connection time. A DNS misconfiguration that causes mild unevenness with stateless HTTP causes catastrophic overload with persistent protocols. (Source: "persistent connections are long-lived and accumulate on whichever origin was resolved at connection time.")

  3. DNS caching obscures the real bottleneck. Even after scaling from 2 to 6 NLB IPs, the problem persisted because Weighted routing returned a single IP per response — all CloudFront edge nodes converged on the same origin for the TTL duration. Adding more origins had no observable effect until the routing policy was changed. (Source: "DNS caching obscured the real bottleneck" section.)

  4. The thundering herd was TTL-driven, not failure-driven. This is a distinct shape from the classic cache-wipe or reconnection thundering herd: the herd forms because all edge nodes resolve to the same origin simultaneously due to DNS TTL synchronization, not because of a failure event. The synchronizer is the DNS TTL window itself. (Source: "The TTL-driven traffic concentration" section.)

  5. The fix was zero-code, zero-architecture-change. bitdrift didn't need to redesign their system — a Route 53 configuration change from Weighted to Multi-Value Answer routing was sufficient to handle 121M devices. This is a powerful example of how a single infrastructure knob can be the determining factor at extreme scale. (Source: "The fix was purely configuration: switching from weighted to multi-value answer routing in Route 53.")

  6. Multi-Value Answer routing requires static IPs (Elastic IPs on NLBs), not Alias records. This is a practical constraint — ALBs cannot use this pattern because they don't support static IP addresses. NLBs with per-AZ Elastic IPs are the required substrate. (Source: Step 3 technical walkthrough.)

  7. Health checks are built-in per record with Multi-Value Answer routing. Unhealthy origins are automatically removed from DNS responses, providing automatic failover that Weighted routing's single-IP-per-response cannot. (Source: Step 2 of the walkthrough.)

  8. Quantified impact: Before (Feb 27, 14M devices): ~80% HTTP 500 errors. After (Mar 8, 121M devices): zero server-side errors. Peak 5xx error rate: 79.80% → 0.033% (2,418× improvement). Average 5xx: 1.87% → 0.003% (623× improvement). (Source: improvement summary table.)

Operational numbers

Metric Value
Peak unique devices (single event) 121 million
Peak requests/sec 110K+
Traffic surge ratio ~100× (near-zero to peak in seconds)
Server-side errors after fix Zero
Before-fix peak 5xx rate (Feb 27) ~80%
After-fix peak 5xx rate (Mar 8) 0.033%
Error rate improvement 2,418× (peak), 623× (average)
DNS TTL used 60 seconds
IPs returned per Multi-Value Answer query Up to 8
NLB origin count 6
Customer-side change DNS configuration update only

Architecture

Mobile devices (121M)
  → Amazon CloudFront (edge cache/proxy)
    → Route 53 DNS resolution (origin domain)
      → Multi-Value Answer: returns up to 8 IPs with health checks
        → NLB-1 (AZ-1) → EKS workloads
        → NLB-2 (AZ-2) → EKS workloads
        → NLB-3 (AZ-3) → EKS workloads
        → NLB-4 ... NLB-6

The failure mode under Weighted routing: Route 53 returned 1 IP per query → all CloudFront cache hosts resolved to the same NLB → all 121M gRPC connections accumulated on that single NLB for the TTL window → catastrophic overload.

Extracted systems

  • systems/amazon-cloudfront — CDN edge absorbing 110K+ RPS from 121M mobile devices; origin resolution via Route 53.
  • systems/amazon-route53 — DNS routing layer between CloudFront and multi-NLB origin fleet; routing policy selection is load-bearing at this scale.
  • systems/aws-nlb — origin fleet of 6 NLBs with Elastic IPs across AZs; target of Multi-Value Answer routing.
  • systems/grpc — persistent connection protocol between mobile SDK and origin; connection accumulation amplifies DNS routing problems.
  • systems/amazon-eks — EKS workloads behind NLBs (briefly mentioned, not elaborated).

Extracted concepts

  • concepts/thundering-herd — new shape: TTL-driven DNS concentration where all CDN edge nodes resolve to the same origin simultaneously.
  • concepts/persistent-connection-amplification — persistent protocols (gRPC, WebSocket) accumulate connections on resolved origins, turning mild DNS unevenness into catastrophic overload.
  • concepts/dns-routing-policy-at-scale — routing policy selection (Weighted vs Multi-Value Answer) has dramatically different behavior at CDN-origin scale; invisible at low scale, decisive at extreme scale.
  • concepts/dns-ttl-concentration — the mechanism: DNS TTL windows synchronize all edge-node resolutions, causing temporal concentration of traffic onto whatever single IP the policy returns.

Extracted patterns

Caveats

  • This is a customer case study with AWS account team involvement, not a pure production retrospective. The narrative has some marketing-lean ("zero errors", "minimal change").
  • No cost numbers, no sustained-load latency distributions, no CloudFront-internal detail on how edge nodes consume multi-value DNS responses.
  • The post doesn't discuss how CloudFront's internal DNS resolution caching interacts with the multi-value responses (do all 8 IPs get used equally? Is there still some temporal skew?).
  • EKS workload details are absent — we don't know the gRPC server architecture, pod count, or how backpressure is handled.
  • The article implies this pattern applies to any CloudFront + multi-NLB + persistent protocol architecture, but the only production validation is gRPC at cricket-event scale.

Source

Last updated · 573 distilled / 1,747 read