Skip to content

PATTERN Cited by 1 source

eBPF header rewrite on egress

Shape: when a packet-header field must hold a value that the control plane hasn't computed yet at tunnel-creation time — but will compute before the first packet flows — attach an eBPF program that rewrites the header field from a placeholder to the real value on egress, and reverses the rewrite on ingress.

The pattern sidesteps kernel APIs that treat the field as immutable post-creation. You pay no kernel-module maintenance burden and the data-plane cost is a few eBPF map lookups + field overwrites per packet.

Canonical instance: AWS Lambda Geneve VNI

AWS Lambda's VPC-mode cold start was bottlenecked by Geneve tunnel creation (~150 ms) because the Virtual Network Identifier (VNI) identifying the customer's VPC isn't known until function init runs inside the micro-VM — and Linux has no API to update a Geneve tunnel's VNI after creation. See concepts/geneve-tunnel-vni.

The fix:

  1. Pre-create tunnels with dummy VNIs while the tunnel is pooled.
  2. When function init produces the real VNI, populate an eBPF map keyed by slot → real VNI.
  3. On packet egress, an attached eBPF program rewrites the Geneve header's VNI field from dummy → real.
  4. On packet ingress, a symmetric eBPF program reverses the rewrite (real → dummy) so the rest of the host stack sees the tunnel it expects.

Result: tunnel latency 150 ms → 200 μs (~750×), tunnel creation moved off the cold-start hot path entirely. (Source: sources/2026-04-22-allthingsdistributed-invisible-engineering-behind-lambdas-network.)

Why eBPF and not:

  • A custom kernel driver — Lambda rejected this to avoid "maintaining Lambda-specific patches upstream indefinitely." See patterns/upstream-the-fix.
  • DPDK — more overhead than eBPF and less in-kernel integration.
  • A user-space proxy — adds a hop and breaks zero-copy.
  • Dynamic tunnel recreation on init — the original problem; it's what Lambda was trying to eliminate.

Generalizes to

  • VXLAN VNI (same architectural shape, different header layout).
  • MPLS labels when a label-switched path is pre-provisioned but the final label is late-bound.
  • IPv6 flow-label assignment in ECMP-friendly load balancing.
  • IPsec SPI context rewrites in SA pooling scenarios.
  • NAT port-mapping — see the sibling pattern where eBPF replaces stateful conntrack-based NAT with predetermined-mapping header rewrite (concepts/stateless-nat-via-ebpf).

Prerequisites for adoption

  1. The placeholder value must route through the stack correctly for packets that are never supposed to reach the wire (during pool setup). Typically you ensure the egress rewrite runs before any packet leaves the host.
  2. The eBPF-map update from control-plane to data-plane must be atomic relative to in-flight packets — usually true for bpf_map_update_elem with hash maps.
  3. The program must pass the eBPF verifier — bounded loops, bounded memory access, termination proofs. See concepts/ebpf-verifier.
  4. You have reviewed the security posture — Lambda was "among the first in Lambda to use eBPF in production" and explicitly notes "there were real questions about whether it would hold up at scale and pass the security reviews that came with it." The existence proof of Cilium helped.

Seen in

Last updated · 319 distilled / 1,201 read