direct preference optimization
Direct Preference Optimization (DPO) is a stable, reward-model-free alternative to RLHF that directly optimizes LLM policy on preference data — achieving comparable alignment results without the complexity and instability of PPO training.
The Problem DPO Solves
- RLHF requires training a separate reward model, then running PPO (complex, unstable).
- PPO has many failure modes: reward hacking, KL explosion, mode collapse.
- DPO (Rafailov et al., 2023) shows: the optimal policy under RLHF has a closed-form relationship to the reward — no need to train the reward model explicitly.
DPO Objective
$$L_{DPO} = -E_{(x,y_w,y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$$
- $y_w$: preferred (winning) response, $y_l$: rejected (losing) response.
- $\pi_{ref}$: SFT reference model (frozen).
- Increase probability of preferred response relative to reference; decrease rejected.
DPO Variants
- IPO (Identity Preference Optimization): Addresses overfitting in DPO.
- KTO (Kahneman-Tversky Optimization): Uses non-paired preference data.
- SimPO: Reference-model-free, uses sequence length normalization.
- ORPO: Combines SFT and preference learning in one stage.
DPO vs. RLHF Comparison
| Aspect | RLHF (PPO) | DPO |
|---|---|---|
| Reward model | Required | Not needed |
| Training stability | Lower | Higher |
| Hyperparameter sensitivity | High | Low |
| Performance | Slightly higher | Close/comparable |
| Implementation complexity | High | Low |
Adoption: DPO is now widely used — Llama 2, Mistral, Gemma, and most open-source aligned models use DPO or variants.
DPO is the standard preference alignment method for open-source LLMs — its simplicity and stability democratized alignment beyond large labs with PPO infrastructure.