DPO (Direct Preference Optimization)
Direct preference optimization without a reward model
DPO (Direct Preference Optimization) was proposed by Stanford in 2023, mathematically redefining the essence of RLHF. While RLHF requires two stages β reward model training β PPO reinforcement learning β DPO directly updates LLM weights from preference data (good response vs. bad response pairs). The key insight is that "the optimal reward function can be derived inversely from the optimal policy." Mathematically, RLHF's objective function reduces to a simple classification loss (cross-entropy). Implementation is much simpler and more stable than RLHF, with comparable or better performance, leading to rapid adoption.
Key Concepts
Collect preference data: prompt + chosen response + rejected response pairs
Freeze Reference Model: use a copy of the LLM before training as the reference model
Compute DPO loss function: log(Ο(chosen)/Ο_ref(chosen)) - log(Ο(rejected)/Ο_ref(rejected))
Update weights to increase probability of chosen responses and decrease probability of rejected responses
Control divergence from reference model with Ξ² parameter
Alignment completed directly without a separate reward model/PPO
Pros
- ✓ No reward model training needed β simplified pipeline
- ✓ More stable training than PPO
- ✓ High memory/compute efficiency
- ✓ Performance comparable to or better than RLHF
Cons
- ✗ Highly sensitive to data quality
- ✗ Cost of building preference pair data
- ✗ Ξ² hyperparameter tuning required
- ✗ Limitations in expressing complex preference patterns