Skip to content

PATTERN Cited by 1 source

Two-stage forecast + optimisation pipeline

Problem

Decision-making systems under uncertainty (inventory / replenishment / capacity / pricing) have two distinct ML workloads with different cost structures and cadences:

  1. A forecasting workload — expensive, model-heavy, produces probabilistic predictions over a horizon.
  2. An optimisation workload — cheaper per-run, consumes forecasts plus current state to find the best decision.

Running both as one monolithic pipeline means every optimisation re-runs the forecast; running them with inconsistent state means optimisation operates on stale or mismatched inputs.

Pattern

Decouple the two workloads into two pipelines with independent cadences that share a forecast output contract:

  1. Stage-1 pipeline — forecasting.
  2. Runs on a slow cadence (weekly / hourly / whatever matches the signal's natural refresh rate).
  3. Produces a probabilistic forecast per entity per time-bucket, persisted to storage consumable by stage 2.

  4. Includes drift monitoring of forecast quality.

  5. Stage-2 pipeline — optimisation.

  6. Runs on a fast cadence (daily batch + real-time online), or continuously.
  7. Consumes the stage-1 forecast plus current state (inventory, price, lead times, stock in transit) as a feature vector per entity.
  8. Runs the optimiser (typically Monte Carlo + gradient-free black-box).
  9. Includes drift monitoring of optimisation performance.

Why it works

  • Cadence decoupling matches real workloads. Forecasts don't need to refresh as often as state — weekly forecast
  • daily optimise + online what-if is the natural cadence for inventory problems.
  • Reusable forecast. One probabilistic forecast can drive replenishment optimisation, pricing optimisation, capacity planning — the team-boundary between the two pipelines maps naturally onto these consumer relationships.
  • Cost control. The expensive stage (forecast) runs less often; the cheap stage (optimise) can iterate fast.

Canonical instance (Zalando ZEOS)

"We break the inventory optimisation problem into two isolated but connected building blocks: Demand Forecast and Inventory Optimisation. The Demand Forecast pipeline is a batch prediction pipeline that produces probabilistic forecasts for articles at a weekly cadence. The Inventory Optimisation pipeline offers daily batch predictions, as well as a real-time inference endpoints to enable our B2B partners to interactively plan inventory settings."

Stage 1 → systems/zeos-demand-forecaster (weekly, 5M SKUs, LightGBM + Nixtla, 12-week horizon, < 2h wall-clock).

Stage 2 → systems/zeos-replenishment-recommender (daily batch SageMaker Batch Transform + online SQS/Lambda, Monte Carlo + black-box optimiser).

Parent system: systems/zeos-inventory-optimisation-system.

Seen in

Last updated · 501 distilled / 1,218 read