Skip to content

PATTERN Cited by 1 source

Dynamic environment composition

Compose a new execution environment at flow-run time from the environments of upstream flows plus additional dependencies of the current task — then resolve that environment at step-execution time, not deploy time. Lets a higher-order flow (e.g. an explainer trainer) operate on arbitrary upstream model environments without anyone pre-declaring the cartesian product.

Problem

You have N upstream flows, each producing a model with its own dependency set. You want a higher-order flow that, given any upstream model, (a) runs inside that model's training environment AND (b) adds dependencies it needs for its own job.

Concrete case from Netflix: training an explainer model per trained model. The explainer flow needs:

  1. The original model and its training environment.
  2. Dependencies specific to building the explainer.

With standard @conda / @pypi, the env is resolved at deploy time. That means you'd have to either pre-enumerate every (upstream-model, explainer) combination (explodes) or give up on reproducibility (breaks ML discipline).

Solution

Use portable execution environments that can be fetched at step-execution time:

  1. Higher-order flow's build_environment step calls the metaflow environment CLI to build an env containing:
  2. the upstream model's declared deps (captured from the upstream run's metadata), plus
  3. the additional deps this flow needs (declared in the flow).
  4. The env is assigned a unique name tied to the run ID + model type.
  5. Subsequent steps reference that unique env name — and fetch it at execution time, not deploy time.

The result: the higher-order flow's environment is composed per-run from the upstream env + local deps, and each run is reproducible because the unique env name is recorded.

Canonical example — "Explainer flow" at Netflix

From the post:

"Explainer flow is event-triggered by an upstream flow, such Model A, B, C flows in the illustration. The build_environment step uses the metaflow environment command provided by our portable environments, to build an environment that includes both the requirements of the input model as well as those needed to build the explainer model itself. The built environment is given a unique name that depends on the run identifier (to provide uniqueness) as well as the model type. Given this environment, the train_explainer step is then able to refer to this uniquely named environment and operate in an environment that can both access the input model as well as train the explainer model." (Source: sources/2024-07-22-netflix-supporting-diverse-ml-systems-at-netflix)

Key distinction from standard @conda / @pypi

"Note that, unlike in typical flows using vanilla @conda or @pypi, the portable environments extension allows users to also fetch those environments directly at execution time as opposed to at deploy time which therefore allows users to, as in this case, resolve the environment right before using it in the next step."

Execution-time env fetch is what unlocks dynamic composition. Without it, the env is baked at deploy time and the higher-order flow can't adapt per-run.

Other applicable use cases

  • Model-specific post-processing — fairness analysers, calibration passes, distillation jobs.
  • A/B-test runners that execute models trained by other teams in the originating team's environment.
  • Auto-evaluation pipelines that replay a model in its own training env.

Trade-offs

  • Requires the extension mechanism, which is not on Metaflow's stable API yet.
  • Env-build step adds latency per run (build time of the composed env).
  • Cache / registry for built envs is implicit — if the backing registry disappears, rerunnability disappears too.

Seen in

Last updated · 550 distilled / 1,221 read