RLHF
Reinforcement Learning from Human Feedback
RLHF is a technique popularized by OpenAI's InstructGPT paper and is the core technology behind ChatGPT. It follows a 3-stage pipeline: Pre-training β SFT (Supervised Fine-Tuning) β RLHF. First, SFT is performed with human-written demonstration responses. Then, a Reward Model is trained on data where humans rank multiple model-generated responses. Finally, PPO (Proximal Policy Optimization) updates the LLM to maximize the reward model's score. This process aligns the model to generate "helpful, harmless, and honest" responses.
Key Concepts
Stage 1 SFT: Learn instruction following from human-written demonstration responses
Stage 2 Reward Model training: model generates multiple responses to the same prompt β humans rank them
Train Reward Model on ranking data β to score "good responses" higher
Stage 3 PPO reinforcement learning: LLM generates response β Reward Model scores it β weights updated toward higher scores
KL divergence penalty constrains the model from diverging too far from the original
Iterate until the model converges on generating responses aligned with human preferences
Pros
- ✓ Generates responses aligned with human values
- ✓ Reduces harmful/biased outputs
- ✓ Dramatic improvement in instruction following
- ✓ Can learn hard-to-quantify "quality"
Cons
- ✗ High cost of human evaluation
- ✗ Reward Hacking β gaming the score without true improvement
- ✗ PPO training instability
- ✗ Evaluator biases are reflected in the model