Speculative Decoding
Small model drafts predictions, large model verifies
LLM token generation is autoregressive, producing only one token at a time. Even with spare GPU compute, memory bandwidth bottlenecks limit speed. Speculative Decoding solves this. A fast Draft model (small) generates ฮณ tokens at once, and the Target model (large) verifies them all in parallel in a single forward pass. If correct, they're accepted; if wrong, the Target model re-generates. Output quality remains identical (mathematically guaranteed same distribution) while speed improves. Variants include Medusa and EAGLE.
Key Concepts
Draft Model (small, fast) autoregressively generates ฮณ candidate tokens
Target Model (large, slow) verifies original prompt + Draft tokens in a single forward pass
Verify each Draft token sequentially โ compare with Target's probability distribution
Accept Draft token if it matches Target distribution
Target model re-generates from first mismatching token (reject + resample)
On average, multiple tokens are confirmed per Target forward pass โ 2-3x speed improvement
Pros
- ✓ Identical output quality (mathematically guaranteed same distribution)
- ✓ 2-3x speed improvement
- ✓ No modification to existing models needed
- ✓ Resolves memory bandwidth bottleneck
Cons
- ✗ Draft Model choice affects performance (high reject rate if too different)
- ✗ Additional memory needed for Draft Model
- ✗ Implementation complexity (verification/resampling logic)
- ✗ Reduced effectiveness in batch processing