Mixture of Experts (MoE)
Efficient scaling by activating only needed experts
Mixture of Experts (MoE) is an architecture that decouples model parameter count from actual computation. While dense models use all parameters for every input, MoE uses a Router/Gate to select only the top-k experts per token. For example, Mixtral 8x7B has 47B total parameters but activates only 2 Experts per token, making actual computation equivalent to a 13B model. GPT-4 is also believed to use MoE architecture. It's a key technology for maintaining large model quality while reducing inference costs.
Key Concepts
Input tokens are passed to the Router (Gate Network)
Router computes probability (weights) for each Expert
Select only top-k Experts (typically k=2)
Selected Experts independently process the input
Generate final output by weighted averaging Expert outputs with router weights
Control token distribution across Experts with Load Balancing Loss
Pros
- ✓ Higher performance with same compute (vs. Dense)
- ✓ Easy parameter scaling
- ✓ Each Expert naturally specializes
- ✓ Cost-efficient inference
Cons
- ✗ Memory usage based on total parameters (large)
- ✗ Load imbalance between Experts
- ✗ Training instability (router training difficulty)
- ✗ Communication overhead in distributed training