CONCEPT Cited by 1 source

Async relationship inference

Definition

Async relationship inference is the metadata-graph primitive of deriving cross-system relationships in scheduled background jobs, rather than computing them at ingestion time on the synchronous write path. New entities arrive with only their direct foreign keys; enrichment jobs walk the graph in the background and materialize derived edges as new facts.

"Once entity metadata is persisted in Datomic, scheduled background processes take over to discover and materialize relationships. These enrichment jobs run periodically, scanning for uncached or partially resolved entities (entities that exist only as references without full metadata)." — sources/2026-05-04-netflix-democratizing-machine-learning-building-the-model-lifecycle-graph

The enrichment workflow

From the Netflix MDS post:

1. Identify candidates — find entities marked uncached or with unresolved references.
2. Hydrate relationships — query source-of-truth systems to fetch related entity details (concepts/source-of-truth-hydration).
3. Materialize edges — write discovered relationships back to the fact store.
4. Re-index — trigger search index updates for the affected entities.
5. Mark as enriched — update entity status to prevent redundant processing.

The loop runs periodically, scanning for new candidates each cycle. Newly arrived entities are picked up on the next cycle; already-enriched entities are skipped.

Why async?

"This asynchronous approach allows MDS to handle the computational cost of graph formation without blocking real-time event ingestion. It also enables retry logic and gradual enrichment as new entities become available." — sources/2026-05-04-netflix-democratizing-machine-learning-building-the-model-lifecycle-graph

Three reasons sync inference would be wrong:

1. Enrichment fan-out is unbounded. A new model instance might reference a pipeline run, which references an A/B test cell, which references an A/B test, which references owners — the transitive closure can be hundreds of API calls. Doing this synchronously would block ingestion.
2. Some referenced entities may not exist yet. If a model instance arrives before its pipeline run is announced, the reference is dangling. The async loop will pick up the relationship later when the pipeline run arrives. A sync approach would either block ingestion or write incomplete state.
3. Retry is natural. If a source system is down, the enrichment job retries on the next cycle. A sync approach would either fail the ingestion event or require an inline retry mechanism.

The staleness budget

Async enrichment has a built-in lag:

"Because enrichment is asynchronous, newly discovered relationships may appear with a short delay after the underlying entities are created (typically minutes rather than seconds)." — sources/2026-05-04-netflix-democratizing-machine-learning-building-the-model-lifecycle-graph

The honest framing: surface the staleness to users.

"We track when each entity was last enriched and surface this timestamp in the AIP Portal, so practitioners can reason about staleness and know when it's safe to rely on a particular relationship for debugging or impact analysis." — sources/2026-05-04-netflix-democratizing-machine-learning-building-the-model-lifecycle-graph

This makes the freshness gradient a queryable property of the graph rather than hidden internal state.

Distinct from inline computation

Aspect	Inline (sync)	Async inference
Latency to ingest	Blocked on full graph walk	Fast — only direct refs stored
Latency to "edge visible"	Zero	Minutes
Failure mode	Ingestion fails or writes incomplete state	Edges appear later; retry on next cycle
Computational cost on ingest path	High	Low
Operational cost on enrichment path	None (no separate job)	New job system to operate
Best for	Low-cardinality, low-fanout graphs	Complex multi-system graphs

Async inference is the right shape for catalog / lineage / metadata services where:

• The graph is large and changes asynchronously across many systems.
• The cost of an "edge appears in 3 minutes" lag is acceptable.
• Sync graph walks would amplify ingestion latency unacceptably.

Distinct from CDC pipelines

CDC also has an async character (replication lag), but it operates on the same data in two places. Async relationship inference operates on new data derived from existing data — it's a graph-level abstraction above the CDC layer.

Seen in

• sources/2026-05-04-netflix-democratizing-machine-learning-building-the-model-lifecycle-graph — Netflix MDS uses async enrichment jobs to materialize multi-hop relationships across six source systems into reified edges in Datomic. The canonical worked example: deriving a direct Model Instance ↔ A/B Test edge from the chain Model Instance → Pipeline Run → A/B Test Cell → A/B Test.