Knowledge Distillation
Transferring knowledge from large models to small models
Knowledge Distillation was proposed by Hinton in 2015. The key idea is that a Teacher model's "soft labels" (probability distributions) contain relationship information between classes beyond just the correct answer. For example, if a Teacher outputs "cat 90%, tiger 8%, dog 2%" for a cat image, it includes the information "cats and tigers are similar." The Student model absorbs the Teacher's "Dark Knowledge" by learning these soft labels. In LLMs, training small models on GPT-4 responses is representative โ Phi-3 and Orca were created this way.
Key Concepts
Train the Teacher model (large) first
Collect Teacher's output probability distributions (soft labels) for training data
Soften probability distribution with Temperature (ฯ) parameter โ maximize information transfer
Student model (small) learns from both soft labels and hard labels (ground truth)
Distillation Loss: KL-Divergence(Student output, Teacher soft label)
Student absorbs Teacher's reasoning patterns (Dark Knowledge)
Pros
- ✓ Small models achieve near-large-model performance
- ✓ Dramatic reduction in inference cost/latency
- ✓ Deployable on mobile/Edge
- ✓ Transfers Teacher's "Dark Knowledge"
Cons
- ✗ Performance ceiling compared to Teacher model
- ✗ Good Teacher is a prerequisite
- ✗ Temperature tuning required
- ✗ Possible API Provider ToS violation (training on GPT-4 outputs)