Skip to content

PATTERN Cited by 1 source

Three-layer graph merge on query

Three-layer graph merge on query is the architectural pattern of building multiple physically separate graphs from independent capture substrates, storing each in the storage shape that best fits its access pattern, and merging them only at query time via parallel traversal + result union — instead of fusing into a single graph at write time.

The wiki's canonical instance is Netflix's Service Topology — three layers (eBPF flows, IPC metrics, end-to-end traces) → three storage substrates (graph DB partition, graph DB partition, columnar storage) → unified view via parallel traversal across all three.

Pattern shape

   ┌──────────────────────────────────────────────────────────┐
   │  Layer 1 capture     Layer 2 capture     Layer 3 capture │
   │  (e.g. network)      (e.g. application)  (e.g. request)  │
   └──────────────────────────────────────────────────────────┘
            │                    │                    │
            ▼                    ▼                    ▼
   ┌──────────────────┐  ┌──────────────────┐  ┌──────────────┐
   │  Storage A       │  │  Storage B       │  │  Storage C   │
   │  (e.g. graph-DB  │  │  (e.g. graph-DB  │  │  (e.g.       │
   │   partition)     │  │   partition)     │  │   columnar)  │
   └──────────────────┘  └──────────────────┘  └──────────────┘
            │                    │                    │
            └─────┬──────────────┴────────────────────┘
                  │      Query-time parallel traversal
                  ▼
        ┌────────────────────────────┐
        │  Result union + merge      │
        │  (preserves per-layer      │
        │   distinct properties)     │
        └────────────────────────────┘
                  │
                  ▼
        Unified view (with optional
        per-layer toggles)

When to use

  • The system needs to answer questions over the union of data sourced from multiple substrates.
  • Each substrate has a distinct blind spot the others compensate for (concepts/multi-source-topology-fusion).
  • Each substrate has a different ideal storage shape — graph DB for path traversal vs columnar for analytical aggregation.
  • Independent layer evolution is operationally important — different substrates have different teams, schema cadences, and upgrade timelines.
  • Per-layer queries are also a useful access mode — engineers may want to look at only the network layer or only the trace layer for some questions.

When NOT to use

  • A single substrate genuinely captures everything — no blind spots → no need for fusion.
  • Query-time latency budget can't tolerate parallel traversal across N storage systems.
  • Storage cost of duplicating edges across layers is unacceptable.
  • Cross-layer joins on per-row attributes are required (parallel traversal + result union is good for graph-shape merge, weaker for per-row joins).

Mechanics

Capture stage

Each substrate runs its own pipeline to its own store. They share only the entity-naming convention (so "App A" in the network graph is the same entity as "App A" in the IPC graph). They do not share schema or storage backend.

Storage stage

Pick the storage shape per layer:

  • Graph database partition for layers whose primary access pattern is graph traversal (multi-hop walks, neighbourhood queries, blast-radius computation). Netflix's network and IPC layers ride Netflix's graph database.
  • Columnar storage for layers whose primary access pattern is analytical aggregation over many edges/events. Netflix's tracing layer rides "columnar storage optimized for analytical queries."
  • Independent partitioning per layer — "physical separation allows each layer to evolve independently and be queried in parallel."

Query stage

  • Single-layer query: dispatch directly to the relevant store; no merge.
  • Unified-view query: fan out the same logical query to all N layers in parallel; collect per-layer results; union nodes and edges while preserving each layer's distinct properties (e.g. an edge present in both the network and IPC layer is annotated with both, not collapsed into a single property bag).

The Netflix post:

"When users request a unified view, we execute traversal queries across all layers simultaneously and merge results, achieving sub-second response times even when combining all three layers." (sources/2026-05-29-netflix-from-silos-to-service-topology-why-netflix-built-a-real-time-service-map)

Why query-time merge over write-time fusion

Three reasons:

  1. Different storage shapes. A write-time fusion forces a uniform store; query-time fusion lets each layer pick.
  2. Independent evolution. Each substrate evolves on its own schedule. A write-time fusion couples them.
  3. Per-layer access remains a first-class mode. Engineers can ask "show me only the network layer" — that's only easy if the layers are still distinct under the hood.

The cost is query-time fan-out latency. Netflix mitigates this with parallelism (all N traversals run simultaneously) and with storage substrates tuned for low-latency multi-hop access; the "sub-second" response-time guarantee is the load-bearing correctness check that the pattern is operationally viable.

Sibling instances on the wiki

  • patterns/dual-store-graph-plus-search-index — sibling at the graph-data-substrate level: two stores, one graph + one search index, with the search index fed asynchronously from the graph store. Different in that the second store is derived from the first; the three-layer pattern has truly independent capture.
  • concepts/multi-source-topology-fusion — the concept-level framing of why-three-graphs.

Generalisation beyond service topology

The pattern is independent of the topology / observability domain: any time you have multiple complementary capture substrates that want to answer queries over their union but have different storage and evolution requirements, the pattern applies. Other domains where it could fit:

  • Multi-modal embeddings + retrieval — independent per-modality vector stores merged at retrieval time.
  • Multi-source identity — each identity-attestation source (corporate IdP, contractor IdP, partner federation) in its own store, merged at policy-decision time.
  • Multi-tier asset inventory — cloud-API-discovered, scanner- discovered, agent-reported assets in independent stores, merged at security-query time.

The shared structural property: complementary substrates with divergent storage shapes + frequent need to answer union-of-substrates queries.

Seen in

Last updated · 542 distilled / 1,571 read