Skip to content

CONCEPT Cited by 2 sources

Exposure bias (ML)

Definition

Exposure bias (in the ML-serving sense) is the production hazard where training data and labels are systematically shaped by the production model's decisions — so a new treatment model being A/B-tested sees a world optimized around the control it's trying to replace.

Concretely: the production model picks which items get shown, which get clicked, which get converted on. Labels (was this ad clicked? did this session convert?) only exist for items the production model decided to expose. Downstream systems — ranking metrics, bid landscapes, user expectations — also equilibrate to the control model's behavior. When a treatment is A/B-tested at a small share (say 20% of traffic), the remaining 80% of the environment is still shaped by control, and the treatment may appear worse than it would under full-production conditions.

This is one of the commonly-raised candidate causes of online-offline discrepancy — but as Pinterest's 2026-02-27 retrospective shows, it often turns out not to be the main cause, and can be cleanly ruled out with a simple methodology.

How to test for it (Pinterest's methodology)

From sources/2026-02-27-pinterest-bridging-the-gap-online-offline-discrepancy-l1-cvr:

"We ran a ramp where treatment traffic went from ~20% up to ~70%, and monitored online calibration and loss for both auction candidates and auction winners before and after the ramp. If exposure bias were the main issue, we would expect treatment metrics to improve as it owned more traffic. We did not see that pattern; the over-calibration issue persisted even at higher treatment shares."

The falsifiable prediction: if exposure bias is dominant, online treatment metrics should improve as its traffic share grows, because the environment becomes progressively more shaped by treatment instead of control. If metrics stay flat (or worse) across a meaningful share ramp, exposure bias is not the dominant cause.

A general recipe:

  1. Ramp treatment traffic share across a meaningful range (at least ~3× — e.g., 20% → 70%).
  2. Compare the target online metric (CPA, CTR, calibration, loss) before and after the ramp.
  3. Check whether there's a treatment-share-responsive trend.
  4. Ramp shows a trend → exposure bias contributes; consider full-launch evaluation or longer soak at higher share.
  5. Ramp shows no trend → exposure bias is not the dominant cause; look elsewhere (feature parity, embedding skew, funnel recall).
  • concepts/self-approval-bias — a related but distinct production hazard where a system evaluates itself against signals it produces, amplifying initial errors.
  • Closed-loop recommendation feedback. The canonical long-term concern: production models train on data they generated, filtering out counterfactuals. Exposure bias is the acute A/B-time manifestation; closed-loop bias is the chronic across-versions manifestation.
  • Propensity weighting / IPS (inverse propensity scoring) is the classical statistical correction for exposure bias in off-policy evaluation — not addressed in the Pinterest post (which found exposure bias wasn't the cause, so no correction needed).

Why it's often not the cause

Pinterest's data point adds empirical weight to treating exposure bias as a hypothesis to rule out, not a default explanation. Common confounders that masquerade as exposure bias but aren't:

  • Feature parity gap between training and serving artifacts (Pinterest's actual cause #1).
  • Embedding version skew between towers in two-tower systems (Pinterest's actual cause #2).
  • Funnel-recall saturation — downstream systems being at their ceiling regardless of treatment share.
  • Metric mismatch — offline metric and online metric measuring different things.

Treating exposure bias as the explanation without running the share-ramp test risks wasted investigation + misdirected engineering effort. A 3-day share ramp + metric-drift check is cheap insurance.

Seen in

  • sources/2026-02-27-pinterest-bridging-the-gap-online-offline-discrepancy-l1-cvr — Pinterest explicitly ruled out exposure bias as the cause of L1 CVR O/O discrepancy by ramping treatment 20% → 70% and observing that the over-calibration issue persisted at higher shares. Canonical wiki example of the ramp-and-compare methodology.
  • sources/2026-04-07-pinterest-evolution-of-multi-objective-optimization-at-pinterest-home — Pinterest Home Feed's diversity-ablation retrospective: removing feed diversification produced a >2% time-spent-impression drop within week 1, with day-1 engagement wins reversing by week 2. Pinterest's explicit mechanism statement: "when users engage with less diverse content, engagement signals will also be affected, reinforcing the system to generate less diverse content." Canonical wiki instance of the closed-loop feedback variant — exposure bias here isn't the acute A/B-window dynamic (which the 2026-02-27 post covers) but the chronic equilibrium drift when a diversity guardrail is removed: subsequent training data systematically tilts toward the less-diverse distribution the production model creates, collapsing variety further. The short-term-vs-long-term divergence makes this mechanism visible only across multi-week soak.
Last updated · 319 distilled / 1,201 read