Skip to content

PATTERN Cited by 1 source

Embedded routing header as address

Intent

Use a large address space (typically IPv6, 128 bits) to encode routing information directly into the address so the forwarding plane can make decisions with static routes against a small number of prefixes, avoiding the operational cost of a dynamic distributed-routing protocol (BGP / OSPF / gossip) over a constantly-churning fleet.

When to use

  • Your fleet has continuously-changing membership (workloads are created and destroyed faster than a routing protocol can converge).
  • You control both ends of the addressing (no external customers pinning addresses).
  • You have DNS or some other naming layer that endpoint consumers can use instead of raw addresses.
  • The cost of running a distributed routing protocol (protocol convergence time, BGP session count, operator expertise) is prohibitive.

Structure

  1. Define an address schema that carves the IPv6 address into routing fields + identifier fields. Fly's 6PN packs "routing information to specific worker servers" into the upper bits of the address and the endpoint identifier into the lower bits.
  2. Install static routes on the forwarding plane for each routing-field prefix → destination worker.
  3. When a workload is created, allocate an address that encodes its routing destination.
  4. When a workload moves, allocate a new address and expose it via the name → address layer (DNS).

Known uses

  • Fly.io 6PN (2024) — Canonical wiki instance. The Making Machines Move post is the explicit retrospective on the migration cost.
  • Carrier-grade IPv6 deployments use similar patterns at the prefix-allocation tier (ISP embeds customer-identity into the delegated prefix), though usually less aggressively than 6PN.

Consequences

Upsides:

  • No distributed-routing-protocol operational burden. "The best routing protocol is 'static'."
  • Scales to very large fleets with constant membership churn.
  • Cheap forwarding — prefix-based static lookup, no runtime protocol overhead.

Downsides:

  • Addresses are not stable across migration. Moving a workload requires allocating a new address and updating the name layer.
  • Reliance on the name layer is absolute. Anyone who bypasses DNS and uses literal addresses breaks on migration — see concepts/hardcoded-literal-address-antipattern.
  • Post-hoc discovery of literal-address users is painful. Fly.io's Postgres-cluster experience: "somebody did use literal IPv6 addresses. It was us." Recovery required both a guest-side compatibility layer and a fleet-wide config rewrite.

Recovery kit

When you discover literal-address use:

  • Ship an in-guest address-mapping compatibility feature first to preserve reachability while you fix configs (patterns/feature-gate-pre-migration-network-rewrite).
  • Then do the fleet-wide rewrite at your leisure.
  • Post-incident, rename or deprecate the literal-address API surface so users can't accidentally repeat the mistake.

Seen in

Last updated · 200 distilled / 1,178 read