Skip to content

PATTERN Cited by 1 source

Multi-value answer routing for origin distribution

Problem

When a CDN (CloudFront) resolves an origin domain via DNS, all edge nodes that resolve during the same TTL window get the same answer. If the routing policy returns a single IP (Weighted routing), all edge-to-origin traffic concentrates on one load balancer per TTL window — creating a thundering herd on that single origin. Adding more origins doesn't help because the routing policy's 1-IP-per-response semantic is the bottleneck, not capacity.

This problem is invisible at low scale or with short-lived HTTP, but becomes catastrophic when combined with persistent-connection protocols (gRPC, WebSocket) that accumulate connections on the resolved origin.

Solution

Use Route 53 Multi-Value Answer routing: returns up to 8 IP addresses per DNS query, each associated with an independent health check. CDN edge nodes can immediately spread connections across multiple origins from the very first DNS resolution, eliminating the single-origin bottleneck.

Structure

Route 53 Hosted Zone (origin.example.com)
├── Record: A / Multi-Value Answer / nlb-1 → 10.0.1.100 [Health: ✓]
├── Record: A / Multi-Value Answer / nlb-2 → 10.0.2.100 [Health: ✓]
├── Record: A / Multi-Value Answer / nlb-3 → 10.0.3.100 [Health: ✓]
├── ...up to 8 records
└── TTL: 60s

DNS response returns all healthy IPs simultaneously → client (CloudFront edge) distributes connections across them.

Constraints

  • Requires IP address records, not Alias records. Multi-Value Answer routing uses ResourceRecords (raw IPs), not AliasTarget. This means NLBs must have Elastic IPs assigned per Availability Zone. ALBs are incompatible (no static IP addresses).
  • Maximum 8 IPs per response. If you have more than 8 origins, only 8 will be returned per query. For most CDN-origin architectures, 8 is sufficient.
  • Health checks are per-record and mandatory. Each record must have an attached health check; unhealthy records are automatically excluded from responses.
  • No weight control. Unlike Weighted routing, you cannot bias traffic toward specific origins. Distribution depends on client-side behavior.

Production reference

bitdrift's T20 World Cup telemetry (Source: sources/2026-07-15-aws-bitdrift-scaled-121-million-grpc-connections-cloudfront): - 121M concurrent gRPC connections via CloudFront → 6 NLBs. - Before (Weighted): single IP per response → 80% 5xx at peak. - After (Multi-Value Answer): all 6 IPs per response → zero 5xx. - Change: DNS configuration only. No code, no architecture change.

When to use

  • CloudFront (or any CDN/edge fleet) with multiple origin load balancers.
  • Persistent-connection protocols (gRPC, WebSocket) where connections accumulate rather than redistribute.
  • Live-event / surge-traffic patterns where traffic goes from near-zero to peak in seconds.
  • Workloads where origin health varies and automatic DNS-level failover is desired.

Seen in

Last updated · 573 distilled / 1,747 read