CONCEPT Cited by 1 source
Reward-weighted training¶
Definition¶
Reward-weighted training is an alignment technique where each training example's loss is scaled by a scalar weight derived from one or more reward models. High-value examples receive larger weights; low-value ones are down-weighted. This biases the model toward behaviors that reward models score highly without the complexity of full reinforcement learning.
Why It Matters¶
Full RL (PPO, GRPO, etc.) requires: - A policy model generating rollouts - A critic/value model for advantage estimation - Complex orchestration of generation + scoring + optimization loops - Significant additional GPU cost
Reward-weighted training avoids all of this — it's a supervised training loop with non-uniform example weights. The reward model runs once offline to label examples; the training loop simply reads the scalar weights. This is simpler, more cost-efficient, and provides effective alignment in practice.
Mechanics (Netflix GenRec)¶
Each training example receives a composite weight from two signal types:
- Long-term satisfaction proxies — estimate how much a short-term engagement contributes to retention, catalog exploration, or sustained engagement.
- Behavior rebalancing — adjust across content types (games vs. movies, new releases vs. evergreen) and launch stages to align with business goals.
The ranking loss for each example is scaled by this weight: loss_i = weight_i × ranking_loss_i.
Trade-offs¶
| Property | Reward-weighted | Full RL (GRPO/PPO) |
|---|---|---|
| Complexity | Low (supervised loop + offline reward scoring) | High (generation + critic + policy optimization) |
| Compute cost | Low | High (2-4× training cost) |
| Alignment quality | Effective in practice | Potentially stronger (on-policy) |
| Freshness | Offline reward labels (can become stale) | On-policy (self-correcting) |
| Implementation | One-line change to loss | Orchestration infrastructure |
Seen in¶
- sources/2026-07-30-netflix-genrec-towards-llm-native-recommendation — "This reward-weighted approach is simpler and more cost-efficient than full reinforcement learning, yet provides effective alignment in practice. We have seen additional gains from RL-style methods (e.g., GRPO), but leave them to future work due to their higher cost."