Skip to content

CONCEPT Cited by 1 source

Geneve tunnel VNI

Generic Network Virtualization Encapsulation (Geneve) is a tunneling protocol commonly used in cloud providers to route a micro-VM's traffic to an isolated tenant network (e.g., an AWS VPC). Every Geneve packet carries a Virtual Network Identifier (VNI) that identifies which tenant network the packet belongs to.

The load-bearing constraint

The VNI is set when the tunnel is created and Linux offers no way to update it afterward. This is a kernel-API-level constraint, not a Geneve-protocol limitation.

Consequence: if you want to create tunnels ahead of time (warm-pool style) but the real VNI isn't known until later in the initialization path, you cannot simply "create the tunnel now, fill in the VNI when we know it." You either:

  • Create the tunnel on-demand once the VNI is known (paying the creation latency on the hot path), or
  • Create the tunnel with a placeholder VNI and find some other way to make the real VNI appear on the wire.

AWS Lambda's case

Lambda's VPC-mode cold start was spending ~150 ms on Geneve tunnel creation alone — the VNI identifying the customer's VPC only became known after function init ran inside the micro-VM, so tunnels couldn't be pre-created. (Source: sources/2026-04-22-allthingsdistributed-invisible-engineering-behind-lambdas-network.)

Options the Lambda team considered

  1. Custom Linux kernel driver that exposes a VNI-update path. Works; the team rejected it to avoid "maintaining Lambda-specific patches upstream indefinitely." See patterns/upstream-the-fix.
  2. DPDK — user-space packet processing. Higher overhead than eBPF, and more complex.
  3. eBPF — attach a program that rewrites the Geneve header on egress (dummy VNI → real VNI) and reverses it on ingress. Kernel stays upstream; tunnels pre-create with a dummy VNI; real VNI comes in via an eBPF map at init time.

Option 3 is what Lambda shipped. Tunnel latency dropped from 150 ms to 200 μs (~750×) and tunnel creation moved off the cold-start hot path entirely. See patterns/ebpf-header-rewrite-on-egress.

Why this is a concept, not a trivium

The constraint generalizes: any packet-header field that is fixed at setup time but requires a late-bound value is a candidate for the same dummy + eBPF-rewrite treatment. The Geneve-VNI case is the canonical disclosure, but the shape recurs in VXLAN VNI, MPLS labels, IPv6 flow labels, and IPsec SPI contexts — any time the "real" identifier needs the control plane to have made progress the data plane hasn't yet seen.

Seen in

Last updated · 319 distilled / 1,201 read