Quantization
Lightening models by converting weights to lower precision
Quantization is a technique for lightening models by representing neural network weights and activations with fewer bits. Typical LLMs stored in FP16 (16-bit floating point) โ a 70B model uses ~140GB memory. Quantizing to INT4 (4-bit) reduces this to ~35GB, making it runnable on consumer GPUs. Post-Training Quantization (PTQ) converts without training, while Quantization-Aware Training (QAT) considers quantization during training. Various formats exist โ GPTQ, AWQ, GGUF โ with llama.cpp's GGUF being the most widely used for local execution.
Key Concepts
Analyze weight distribution of the original model (outliers, ranges, etc.)
Determine quantization range โ set intervals for mapping weight values to lower bits
Convert weights to specified bit count (FP16 โ INT4: scale each value to 0-15 range)
Store scale factor and zero point separately (used for dequantization)
Inference: restore approximate values with quantized weights ร scale factor โ compute
Accuracy verification: compare pre/post-quantization performance (perplexity, benchmarks, etc.)
Pros
- ✓ Model size reduced by 1/2 to 1/8
- ✓ Improved inference speed (resolves memory bandwidth bottleneck)
- ✓ Run large models on consumer GPUs
- ✓ Cost reduction (serving infrastructure)
Cons
- ✗ Accuracy loss (especially below 4-bit)
- ✗ Difficulty handling outliers
- ✗ Compatibility issues between quantization formats
- ✗ Severe performance degradation on some tasks