Skip to content

CONCEPT Cited by 1 source

Candidate survival rate

Definition

Candidate survival rate is the share of candidates retrieved by a candidate generator (CG) that survive the downstream ranking funnel and reach actual impression — i.e. are shown to users. It is the load-bearing funnel metric for diagnosing whether a CG is a meaningful contributor to delivered traffic, distinct from intrinsic CG quality measures like recall@K which evaluate the CG in isolation.

Pinterest's canonical framing (sources/2026-05-08-pinterest-enhancing-ad-relevance-integrating-real-time-context-into-sequential-recommender-models):

"Less than 1% of impressions on Related Pins were attributed to this CG, indicating its candidates struggled to survive the downstream ranking and auction stages."

Why survival rate is distinct from recall@K

A CG is a retrieval-stage component: it narrows millions of items to thousands, which then flow into a ranking funnel (L1 ranker, L2 ranker, ads auction, business rules) before any candidate reaches user impression. Two CGs can have similar offline recall@K but very different survival rates:

  • High recall, low survival: the CG retrieves relevant candidates (high recall against held-out positives) but those candidates lose to candidates from other CGs in the downstream funnel — typically because the other CGs see signals (real-time context, surface-specific features, freshness) that this CG doesn't.
  • High recall, high survival: the CG retrieves candidates that the downstream funnel agrees are good — i.e., the CG and the ranker share enough signal to align on which candidates win.

Pinterest's prior offline-only Sequential CG had the high-recall/low-survival pathology on Related Pins: it retrieved candidates correctly against held-out offsite-conversion labels but those candidates didn't reflect the user's current Related-Pins-page intent, so the L1/L2 rankers — which did see subject-Pin context — kept dropping them. The fix wasn't "better retrieval" in isolation; it was giving the CG access to the same intent signal the rankers see, via a context layer.

Survival rate as the operational diagnostic for adding context

The survival-rate metric is what makes the case for context-aware retrieval rigorous. "Our recall@K is fine" is an insufficient defence if survival is collapsing — it just means the CG is being correctly out-competed by ranker-aligned alternatives. Pinterest's improvements:

Metric Before context layer After context layer
Impression share on Related Pins < 1% (not directly stated; 2x candidate delivery)
Median candidate relevance baseline +275–300%
Ads relevance metric baseline +1.08%
Candidate delivery (retrieved → impression) baseline 2x
Recall@K (offline) baseline 3x–10x

Note the recall@K and survival improvements move together — the model is genuinely retrieving better candidates and those candidates are now competitive in the funnel. Both improvements are needed; either one alone wouldn't have moved topline ROAS (~0.7%).

Survival rate as a structural argument for ranker-CG signal alignment

The general design lesson: a retrieval-stage component that doesn't see the same signals the downstream rankers see will be structurally disadvantaged in the funnel, regardless of its intrinsic retrieval quality. The fix is signal alignment:

  • If rankers see real-time context, the CG should see real-time context.
  • If rankers see surface-specific features, the CG should see surface-specific features.
  • If rankers see fresh feature values, the CG can't be entirely batched.

This is one driver behind the hybrid offline/online tower inference pattern: the CG needs online-computed access to the same intent signals the online ranker has, even at the cost of a more complex serving stack.

Generalisation

Candidate survival rate as a metric applies to any retrieval-stage component feeding a ranking funnel:

  • Ads CGs (Pinterest's case): retrieved → ranker → auction → impression.
  • Search retrieval models: retrieved → reranker → result page → click.
  • Recommendation candidate sources: retrieved → blender / multi-objective rank → feed → engagement.
  • Generic two-stage retrieval-rerank pipelines: retrieval → cross-encoder rerank → returned to user.

In each case, the CG owner can locally optimise recall@K all they want; the funnel-level question is what fraction of their candidates survive.

Caveats

  • Pinterest's specific framing is for ads CG → L1 → L2 → auction. Other domains have different funnel shapes (e.g., search has retrieval + rerank but no auction; recommendation has retrieval + blender but typically no auction).
  • Survival is a downstream-driven metric. A CG can improve its survival not by changing its own retrieval but because the downstream ranker improved or another CG degraded. Useful to track survival changes alongside retrieval changes.
  • Calibration matters. "Less than 1% of impressions" could be the correct steady state if other CGs are stronger on this surface. The fact that Pinterest treated it as a problem implies they had reason to believe the CG's intrinsic quality should support a meaningfully higher share.
  • No published target — Pinterest doesn't disclose what survival rate they consider "healthy" for the contextual sequential CG.

Seen in

Last updated · 542 distilled / 1,571 read