πŸ”§

LoRA (Low-Rank Adaptation)

Efficient fine-tuning that trains only a small number of parameters

LoRA (Low-Rank Adaptation) is a PEFT (Parameter-Efficient Fine-Tuning) technique proposed by Microsoft in 2021. Full Fine-tuning updates all model weights (tens to hundreds of billions), requiring massive GPU memory. LoRA's key insight is that "the weight change (Ξ”W) during fine-tuning is low-rank." It freezes the original weight matrix W(dΓ—d) and decomposes Ξ”W = A(dΓ—r) Γ— B(rΓ—d) (r << d, typically r=4-64). Trainable parameters drop to 0.1-1% of the original, and at inference, W + Ξ”W can be pre-merged with zero additional latency. QLoRA combines this with quantization, enabling fine-tuning on 4-bit models.

Key Concepts

1

Freeze the original model's weight matrix W

2

Add low-rank matrices A(dΓ—r) and B(rΓ—d) alongside W (r is rank, typically 4-64)

3

Forward pass: y = Wx + (Ξ±/r) Γ— ABx β€” add LoRA output to original output

4

Backpropagation: only update weights of A and B (W remains frozen)

5

Trainable parameters: 2Γ—dΓ—r (dramatic reduction from original dΓ—d)

6

Inference: merge as W_new = W + (Ξ±/r)Γ—AB β†’ zero additional compute cost

Pros

  • ✓ Massive GPU memory savings (1/3 to 1/10 of Full FT)
  • ✓ Improved training speed
  • ✓ Multi-purpose usage by swapping task-specific adapters
  • ✓ No additional inference cost (can be merged)
  • ✓ Original model preserved

Cons

  • ✗ Performance may be slightly lower than Full Fine-tuning
  • ✗ Rank (r) selection affects performance
  • ✗ Need to decide whether to apply to all layers or only some
  • ✗ Limitations for complex domain shifts

Use Cases

Fine-tuning 70B models on consumer GPUs (RTX 4090) QLoRA β€” 4-bit quantization + LoRA Managing multiple task-specific adapters Hugging Face PEFT library Stable Diffusion LoRA (image style learning)

References

Related

🎯
Fine-tuning
Re-training a pre-trained model for specific tasks
→
← 🎲 DPO (Direct Preference Optimization) ✍️ Prompt Engineering →