Skip to content

CONCEPT Cited by 1 source

Switch-back experiment

Definition

A switch-back experiment is an experimental design that alternates treatment and control over time in the same unit (typically a single market) — e.g. hours 0–2 run policy A, hours 2–4 run policy B, repeat — and compares outcomes across the treatment periods. It is the time-based sibling of classical user-split A/B testing (which splits users) and region-split (which splits geographies).

Switch-back is the experiment shape of choice in two-sided marketplaces when:

  • Market interference breaks user-split SUTVA — treatment and control users sharing the same market affect each other through prices / supply / wait times (market mediation).
  • A whole-market shock is too expensive to run across multiple geographies, or the experiment needs finer time resolution than a region-split can produce.

By alternating policy across time in a single market, a switch-back observes how the market itself responds to each policy setting, at the natural granularity of whatever process is being modelled (an hourly policy can be tested with hour-long slots).

Key failure modes

  • Carryover effects. The state of the market at the end of a treatment slot partly reflects the previous slot's policy (e.g. if treatment A caused drivers to reposition, treatment B inherits that repositioning). Mitigations: wash-out intervals, longer slots, block randomisation with sufficient separation.
  • Time confounding. If slots are assigned non-randomly (e.g. treatment is always 2–4pm), time-of-day effects confound with treatment. Mitigation: randomise slot assignment within comparable time windows.
  • Autocorrelation in outcomes. Slot-level outcomes are not IID; proper standard errors require accounting for serial correlation.
  • Operational disruption. Rapid policy switching may be unacceptable in production for customer-experience reasons; in practice, switch-back is often restricted to carefully chosen markets and carefully chosen policies.

Why Lyft uses it to verify Step 1 of surrogacy

In Lyft's surrogacy framework, Step 1 estimates how policy decisions shift the distribution of short-term negative user experiences. Validating that estimate requires an experiment where the market itself responds to the policy and the response shows up in measured experience — exactly what switch-back produces. Lyft:

"To validate this mapping, we use switch-back experiments that alternate policy settings across comparable time slots and compare the model's predicted changes in negative user experience to the experimental lifts we observe. The experiment either verifies our model or informs its iteration (e.g., changing controls or introducing additional ones)."

What switch-back cannot verify: the long-term outcome half of the surrogacy framework. The time slots are too short for retention or future-rides effects to manifest at the slot level. That's why switch-back covers Step 1 only, with user-split verifying Step 2 and region-split verifying the composed end-to-end forecast.

Comparison

Experiment Randomisation unit Observes market mediation Long-term outcomes
User-split user ❌ no ✅ yes (long windows)
Switch-back time slot ✅ yes (within slot) ❌ no (slots too short)
Region-split region ✅ yes (whole market) ✅ yes

Seen in

  • Lyft — Beyond A/B Testing (2026-03-25) — canonical wiki instance. Lyft uses switch-back experiments to verify the Step 1 (policy → short-term negative user experience) estimator of its surrogacy framework.
Last updated · 319 distilled / 1,201 read