Skip to content

CONCEPT Cited by 1 source

Feature coverage dashboard

Definition

A feature coverage dashboard is the operational instrument that makes the training-serving feature gap visible in production. For each feature family the model expects to use, it shows — in real time — what fraction of live serving requests actually have that feature populated (non-default, non-null, non-zero) at the specific serving artifact the model reads from (feature store, ANN index, embedding cache, etc.).

The dashboard is the ground-truth signal that distinguishes "a feature exists in the system" from "a feature is actually reaching the model at serving time." Pinterest cites it as the primary diagnostic instrument that surfaced their L1 embedding feature gap (sources/2026-02-27-pinterest-bridging-the-gap-online-offline-discrepancy-l1-cvr).

Why it's necessary

A feature can live in three places without being available at serving:

  1. Training logs — the ML team's offline view.
  2. The request-time feature store (e.g., L2 Feature Store at Pinterest) — populated for some serving stages.
  3. The model-specific serving artifact (ANN index, embedding cache, ranker input table) — populated separately, often via a distinct pipeline.

All three views can be out of sync. Without a dashboard that measures coverage at the specific artifact the model reads from, the organization has no reliable way to know:

  • Which features are actually present at serving time vs present only in training.
  • Which feature families regressed when a pipeline changed.
  • Whether a new feature has finished onboarding end-to-end, or is stuck partway through the fanout of pipelines.

The answer to "is feature X available in production?" must be a number, not a belief.

Pinterest's usage pattern

From the 2026-02-27 post:

"When we put the two side by side (offline insertion tables vs. online feature-coverage dashboards), it turned out several high-impact Pin feature families had never made it into the L1 embedding path at all..."

The pattern is:

  1. Offline insertion tables — what the training job + model registry thinks it has.
  2. Online coverage dashboards — what's actually populated in the serving artifact (the L1 ANN index in Pinterest's case).
  3. Diff the two.

Pinterest found whole feature families (targeting-spec flags, offsite conversion visit counts at 1/7/30/90 days, annotations, MediaSage image embeddings) missing from the online side. The fix — onboard them via UFR — was visible on the coverage dashboards as coverage recovered.

This side-by-side comparison is the core of the feature parity audit pattern.

Design principles

  • Measure at the actual serving-read point. A feature reaching the L2 Feature Store is different from a feature reaching the L1 ANN-index build job. Measure where the model actually reads.
  • Granularity matches feature granularity. Per-feature (or per-feature-family) coverage — aggregate "95% of requests have features" is useless for diagnosis.
  • Segment by relevant axes. Per-traffic-slice (oCPM vs CPM, shopping vs non-shopping), per-surface, per-tier. A feature can be 100% covered on one slice and 0% on another.
  • Surface regressions automatically. Alerting on coverage drops catches pipeline regressions before model quality does.
  • Close the loop with onboarding tooling. Pinterest's UFR is the control plane; dashboards are the read side. Coverage-drop alerts should point back at the onboarding config that controls the feature.

Part of the debuggability-as-product discipline

Pinterest's closing frame: "Debuggability is part of the product. Coverage dashboards, embedding skew tests, and parity harnesses are as important to model velocity as the architecture itself."

Feature coverage dashboards sit alongside:

  • Embedding version skew tests.
  • Training-vs-serving parity harnesses (replay a training batch through the online serving stack and check outputs match).
  • Funnel-recall tracking at each stage boundary.

These are treated as first-class infrastructure, not side-investments.

Seen in

Last updated · 319 distilled / 1,201 read