Skip to content

CONCEPT Cited by 1 source

Two-stage forecast + optimise pipeline

Definition

A two-stage forecast + optimise pipeline is an architectural idiom for decision-making systems under uncertainty where:

  1. Stage 1 (forecast) — a machine-learning pipeline produces a probabilistic forecast (or a set of probabilistic inputs) over a prediction horizon.
  2. Stage 2 (optimise) — an optimisation pipeline consumes those forecasts as inputs, together with the current state (inventory, demand signals, prices), and searches for the decision that minimises expected cost (or maximises expected value).

The two stages have separate cadences, separate resource profiles, and separate failure modes — the forecast is expensive and can be refreshed less often; the optimisation is cheaper but needs to reflect the latest state.

Why decouple

  • Different cost / cadence profiles. Forecasts are model-training-heavy and slow to retrain (in Zalando's case, weekly). Optimisation consumes the forecast and recomputes daily (or online on-demand) as state changes.
  • Different skill sets. Forecasting is an MLE / applied science problem; optimisation is an OR / MLE problem. Decoupling maps the org split onto the pipeline split.
  • Reuse of forecasts across consumers. A probabilistic-demand forecast can be consumed by replenishment optimisation, pricing optimisation, capacity planning, marketing — one forecast powers many optimisers.

Canonical instance (Zalando ZEOS)

"We break the inventory optimisation problem into two isolated but connected building blocks: Demand Forecast and Inventory Optimisation. The Demand Forecast pipeline is a batch prediction pipeline that produces probabilistic forecasts for articles at a weekly cadence. The Inventory Optimisation pipeline offers daily batch predictions, as well as a real-time inference endpoints to enable our B2B partners to interactively plan inventory settings."

Stage 1 → systems/zeos-demand-forecaster (weekly, 5M SKUs, LightGBM + Nixtla MLForecast, 12-week horizon).

Stage 2 → systems/zeos-replenishment-recommender (daily batch via SageMaker Batch Transform + online via SQS/Lambda, Monte Carlo + gradient-free black-box optimiser).

Cadence decoupling is load-bearing

Zalando's weekly forecast + daily optimisation cadence split is not arbitrary — the forecast is the expensive workload, so a lower refresh frequency is acceptable; recent changes to inventory / settings / stock can still be captured in the cheaper daily (or online) optimisation layer without re-running the forecast. See patterns/weekly-batch-forecast-daily-batch-optimise-cadence.

Seen in

Last updated · 501 distilled / 1,218 read