Skip to content

CONCEPT Cited by 1 source

Traffic shadowing via ingress

Definition

Traffic shadowing via ingress is the practice of duplicating production requests at the ingress layer and sending the copy to a second backend while the original still serves the user from the primary backend. The user-visible request is unaffected. Different from client-side dual-write (which requires application code changes): the duplication is invisible to both the application and the client.

Canonical mechanism at the ingress:

  1. The primary route matches the request and forwards to the production backend.
  2. A filter on that route duplicates the request (shadow clone) and tags it (e.g. sets a named Tee).
  3. A secondary route matches the tag/predicate and forwards the shadow to the new backend.

The response of the shadow backend is discarded (or logged for comparison); the primary backend's response is returned to the client. This is in contrast to dark-shipping where the duplication happens inside the application process.

Load-bearing properties

  • Write-side only in most use cases — shadowing read traffic gives you latency-and-error parity data but not state-convergence. Shadowing writes convergence the shadow backend with the primary.
  • Application-code-invisible — zero coordination with application teams, can be rolled out by the platform/ops team alone.
  • Mirror-order consistency is not preserved under load — shadow requests may arrive slightly out of order, which matters for writes with non-commutative effects.

Seen in

  • sources/2023-11-19-zalando-migrating-from-elasticsearch-7-to-8 — Canonical wiki instance at Skipper ingress, using the teeLoopback filter + Tee(...) predicate pair. The two Skipper routes are named explicitly in the RouteGroup YAML in the source page: route 1 matches /_bulk / /_alias/{index}_write endpoints on the old ES7 backend and applies teeLoopback("intake_shadow"); route 2 matches the Tee("intake_shadow") predicate and forwards to the ES8 backend. A Weight(2) hack is added to prevent the second route from colliding with a Traffic(...) percentage-split route. Shadowing is applied sequentially: first to template/mapping endpoints (so the new cluster has correct index templates before any data arrives), then to bulk-write endpoints, then to query endpoints for A/B comparison, before the final routing flip.

Siblings across the wiki

  • Client-altitude shadowing — dark-ship (GitHub Issues) runs the new codepath inside the application process, inline on the request, and diff-logs. Different altitude, same goal (behavior parity validation).
  • Offline / batch shadow — patterns/shadow-migration replays a dataset through both the old and new path and compares. Not live-request-path.
  • L4 / envelope-level shadow — Envoy's request_mirror_policies is the same primitive at the L7 mesh layer, not the ingress.

Ingress-level shadowing's distinguishing property is that it is the cheapest to deploy (ingress-config change, no application change) and the hardest to control per-request (predicates only see what the proxy can see: path, headers, method).

Last updated · 501 distilled / 1,218 read