Skip to content

PATTERN Cited by 1 source

Open-source library wrapped by production system

Problem

A new platform needs domain expertise from two or more subgroups whose stacks barely overlap. Classic instance: a data-scientist team (Python, statistics PhDs, Jupyter notebooks, no production-service experience) and a backend engineer team (Scala / JVM, distributed systems, no formal statistics training). Either:

  • The scientists learn the engineers' stack — months lost, scientific quality drops while they ramp.
  • The engineers learn the scientists' domain — months lost, platform quality drops while they ramp.
  • They work serially (scientists hand specs to engineers who reimplement) — specs get mis-translated, two implementations drift.

All three pattern anti-patterns extend the calendar to unworkable and damage both subgroups' ability to produce work in their specialty.

Solution

Have scientists build a standalone open-source library in their preferred language that implements the domain logic (e.g. statistical analysis routines). Have engineers build the production service — including logging, scaling, observability, latency — that wraps the library as a dependency. The library is the shared contract; each subgroup iterates in its own stack against that contract.

Zalando's applied case

At Octopus's founding (pre-2015 — crawl phase of concepts/experimentation-evolution-model-fabijan), the virtual team had engineers and data scientists with "little knowledge of each other's domain" — "data scientists didn't have production software experience and didn't know Scala, whereas software engineers didn't know concepts of statistics" (Source: sources/2021-01-11-zalando-experimentation-platform-at-zalando-part-1-evolution).

The resolution: an open-source statistics library wrapped by the backend production system. The library handles the statistical correctness. The backend handles concurrency, ingestion, API, persistence, alerting, dashboards.

Why "open-source" matters

The article specifically calls out open-source as part of the pattern, not just "a shared library". Open-sourcing:

  • Forces clean API boundaries — no production-private conveniences leaking in that would couple the library to the wrapper.
  • Gets external contribution / review for the domain logic (in Zalando's case, academic + industry stats community).
  • Decouples scientist careers from the employer's code — they build reputation on the library itself.
  • Avoids the worst drift mode: the library becoming a custom fork with quirks only one company understands.

Trade-offs

  • Wrap-layer complexity — the wrapper has to handle everything the library deliberately doesn't: auth, concurrency, error-to-metric conversion, replay, persistence. Expect the wrapper to be non-trivial.
  • Library release cadence — scientists' release cycle (papers, conferences, periodic) will be slower than the production system's. The wrapper must tolerate library API stability + occasional breaking changes.
  • Scaling the library — the library is typically single- process / non-concurrent. Scaling is the wrapper's job (parallel invocation, batching, caching, distribution). In Octopus's case, scaling the analysis system ultimately required rebuilding on Spark after the initial wrapper architecture reached its ceiling — ~2 years of work.

When to use

  • Cross-functional platform where two subgroups' stacks don't overlap (stats + backend, ML researcher + MLE, hardware verification + embedded software).
  • Domain logic is the hard part; production concerns are secondary in complexity.
  • The domain logic is reusable outside this platform — the library stands on its own.

When not to use

  • Domain logic is tightly coupled to the wrapper's state (highly stateful middleware, transactional side effects).
  • A single polyglot team can own both sides — then the open boundary is bureaucratic overhead.
  • The domain is not stable enough to commit to a public API.

Cross-reference

Related but different: patterns/upstream-the-fix is the dual pattern for fixes going out to community; this pattern is about building the library first as the shared contract.

Seen in

Last updated · 476 distilled / 1,218 read