Skip to content

CONCEPT Cited by 1 source

Ablation study — forecast type × objective type

Definition

An ablation study in ML / optimisation systems isolates the contribution of each design choice by running the full system with one component replaced by a simpler baseline, holding everything else fixed. The deltas across ablation cells tell you which component contributes how much to the headline metric.

In inventory-optimisation context, Zalando's paper runs a 2 × 2 ablation:

  • Axis 1: Forecast type — {Probabilistic, Point}.
  • Axis 2: Objective type — {P75 (percentile), Mean}.

The study holds everything else constant: same Extended (R, s, Q) policy, same DES simulator, same gradient-free optimiser, same backtest data. Only the forecast + objective cells change.

Results (verbatim)

Configuration GMV Uplift GMV after FC Availability Fill Rate
Probabilistic Forecast + Percentile Objective (ours) 22.11% 21.95% 86.40% 91.14%
Probabilistic Forecast + Mean Objective 19.02% 20.16% 81.27% 87.98%
Point Forecast + Percentile Objective 6.37% 5.98% 77.76% 84.95%

(The fourth cell — Point + Mean — is typically the textbook baseline and is not reported in the paper's summary; implicitly it's worse than the Point + Percentile row.)

Reading the deltas

Row 1 vs Row 2 — percentile objective's contribution (controlling for probabilistic forecast):

  • GMV: +3.09pp
  • GMV after FC: +1.79pp
  • Availability: +5.13pp
  • Fill rate: +3.16pp

Row 1 vs Row 3 — probabilistic forecast's contribution (controlling for percentile objective):

  • GMV: +15.74pp
  • GMV after FC: +15.97pp
  • Availability: +8.64pp
  • Fill rate: +6.19pp

Zalando's reading

Verbatim: "You need both. Switching from point forecasts to probabilistic ones provides the single largest gain. However, optimizing for the 75th percentile rather than the average provides that final, critical layer of stability, particularly in protecting the merchant against high-impact 'tail' events."

This canonicalises a two-tier lever structure for decision-under-uncertainty systems:

  • First-order lever — probabilistic forecasting. Without tail information about demand, risk-aware optimisation has nothing to act on — the optimiser sees only a point estimate and behaves the same regardless of objective function.
  • Second-order lever — percentile objective. Given a probabilistic forecast, the objective-type choice determines whether the optimiser uses the tail information or averages it out. Mean objective wastes most of the probabilistic information.

The cost order-of-magnitude: forecast type contributes ~5× the incremental lift of objective type, but both are needed to reach the 22% total.

Why the two levers are complementary

  • Probabilistic forecast without percentile objective (Row 2): The optimiser collapses uncertainty by taking the mean of the Monte Carlo samples, losing most of the tail information. Still beats point-forecast baseline because the mean of a probabilistic forecast is a better point estimate than a naive point forecast (which typically has biases from training-data asymmetries).
  • Point forecast with percentile objective (Row 3): The cost distribution collapses to a delta at the point-forecast value — P75(delta) = mean(delta) = point_forecast. The percentile objective has no tail to optimise over. Becomes effectively equivalent to mean objective.
  • Probabilistic forecast + percentile objective (Row 1): Both axes contribute. The probabilistic forecast supplies a distribution; the percentile objective makes use of its tail.

What the ablation doesn't show

  • Sensitivity to percentile choice. P75 vs P90 vs P95 — not ablated in the paper.
  • Sensitivity to forecast quality. The paper uses LightGBM quantile regression. An ablation over forecast model family (LightGBM vs TFT vs DeepAR) is not shown.
  • Interaction with policy family. Whether probabilistic+P75 beats mean+point more or less dramatically with different policy families (classical (s, S) vs extended (R, s, Q)) is not tested.
  • Adoption-curve sensitivity. All ablation cells assume 100% adoption — the lever structure might shift under partial adoption.

Why this decomposition matters for system design

Teams building decision-under-uncertainty systems often face a question: "do we need to invest in probabilistic forecasting, or is risk-aware optimisation enough?"

Zalando's ablation gives the answer: invest in probabilistic forecasting first; add risk-aware optimisation second. The probabilistic forecast is load-bearing; the risk-aware objective amplifies its value. Spending engineering budget on risk-aware objectives on top of point forecasts wastes the effort (Row 3 is barely better than the baseline).

Seen in

Last updated · 428 distilled / 1,221 read