Skip to content

PATTERN Cited by 1 source

Controlled experiment before shipping

Problem

Product teams shipping a UX change that adds a step to a conversion funnel face a recurring risk: the new step may improve the downstream metric (the thing the step is optimising) while regressing the upstream metric (the user completing the funnel at all). If no one measures the upstream regression explicitly, the team ships the change, sees the feature-metric win, and discovers weeks later that total conversion dropped.

The failure mode is optimising for the feature's metric in isolation from the funnel it lives in.

Solution

Before rolling a UX change broadly, run a controlled experiment (A/B test) whose primary metric is not the feature's own metric but the upstream funnel completion metric that the added friction could regress. Ship only if the upstream metric holds.

Optional secondary: measure whether the feature actually produces behavioural change (people use the new option), not just positive sentiment.

Canonical instance — Lyft gated-community pickup launch

Per the 2026-04-23 Lyft write-up (sources/2026-04-23-lyft-smarter-pickup-experience-for-gated-communities):

"Before shipping, we ran a controlled experiment to make sure we weren't accidentally making things worse — specifically, that the new flow wouldn't add enough friction to make riders bail on requesting a ride altogether. Good news: no meaningful drop-off. Even better news: we saw actual pickups happening at the spots that riders chose initially, meaning that they no longer have to walk around to find their drivers."

Two explicit metrics:

  • Primary (upstream): share of requests that complete to dispatch ("riders bail on requesting a ride altogether"). Verdict: no meaningful regression.
  • Secondary (behavioural): actual pickups at chosen spots (not the driver ending up elsewhere). Verdict: rider choice was honoured.

The sentiment metric (~95% positive survey response) was measured post-launch, not in the pre-ship experiment — consistent with the principle that survey sentiment is a rationalisation, not a funnel metric.

Why this shape works

  • Disentangles effect from regression. Without the upstream-metric guardrail, a team might ship and see "pickup-spot selection usage up 40%" while missing "total ride requests down 2%".
  • Forces honesty about tradeoffs. If the UX does introduce meaningful friction, the experiment catches it and forces the team to redesign, not rationalise.
  • Sets the bar at the right place. "No meaningful drop-off" is a bar that passes real features and blocks badly-scoped ones.

Where this applies

Any UX change that:

  • Adds a new step, decision, or screen to a funnel.
  • Replaces a default-on path with a user-selectable path.
  • Adds a confirmation or information-gather step pre-dispatch.

The pattern is cheap to apply (you're already likely to A/B test) and the critical discipline is choosing the upstream metric as primary, not the feature's own metric.

Last updated · 550 distilled / 1,221 read