Skip to content

CONCEPT Cited by 1 source

Experimentation culture

Definition

Experimentation culture is the organisational state in which data-driven decisions via controlled experiments are the default, not an exception: every significant product change is expected to ship through an A/B test or a principled causal inference alternative, and teams maintain the domain knowledge, staffing, and authority needed to run experiments autonomously and trustworthily.

Culture in this sense is distinct from platform capability — an org can have a fine experimentation platform that nobody uses because product leaders don't ask for experiment results before shipping, or because teams lack statistically literate owners. Culture and platform co-evolve (Source: sources/2021-01-11-zalando-experimentation-platform-at-zalando-part-1-evolution).

Why it is hard to build

Zalando's 2021 retrospective is explicit that "data-driven decisions as natural as they sound today" were not the focus in Zalando's early days. Even after the first platform shipped in 2015, many teams still didn't A/B-test before deciding. The barrier was not tooling — it was:

  • Lack of A/B-testing knowledge in product teams — no notion of what a hypothesis, a statistically powered test, or a trustworthy KPI looks like.
  • No embedded experimentation owner per team — nobody with enough statistical fluency to design, launch, and interpret tests for that team's domain.
  • No visibility at leadership level into whether experiments actually happened before decisions.

Zalando's interventions

Octopus's walk-phase lists five moves that together built experimentation culture org-wide:

  1. Integrate use cases from many departments into the single platform so the platform matched the range of real decisions being made.
  2. Regular in-person A/B-testing training — repeated cohorts, not a one-off.
  3. Company-wide initiative to ensure each team has embedded A/B-test owners (product analysts or data scientists) with sufficient experimentation knowledge.
  4. Internal blogs sharing tips for effective A/B testing on Octopus — reinforcing practice at the individual-IC level.
  5. Weekly consultation hours + A/B test design audit process so even teams without strong statistical fluency get a trustworthy test design (see concepts/ab-test-design-audit).

Zalando's Run-phase (2020+) extends this with a new company-wide training curriculum covering causality, statistical inference, and Zalando-specific analysis tools; plus expanding causal-inference research peer reviews from the experimentation team to the whole company.

Relationship to the Evolution Model

In the concepts/experimentation-evolution-model-fabijan, Crawl-phase orgs have no experimentation culture; Walk-phase orgs build it; Run-phase orgs have it and can move attention to advanced statistical methods. Zalando explicitly places itself at Run phase in the 2021 post.

Seen in

Last updated · 476 distilled / 1,218 read