Diffusion Model
Generating images by progressively removing noise
Diffusion Models are inspired by thermodynamic diffusion processes. The Forward Process gradually adds Gaussian noise to the original image until it becomes pure noise. In the Reverse Process, a neural network (U-Net or DiT) predicts and removes noise at each step to restore the image. During training, the model learns to predict added noise at arbitrary noise levels. Text-conditional generation injects CLIP text encoder embeddings via Cross-Attention. Stable Diffusion performs diffusion in Latent Space, reducing computation by 64x.
Key Concepts
Forward Process: gradually add Gaussian noise to original image xβ over T steps β xT (pure noise)
Training: train U-Net/DiT to predict noise Ξ΅ added at arbitrary step t
Text conditioning: CLIP text encoder embeds prompt β inject into U-Net via Cross-Attention
Reverse Process: starting from xT (pure noise), remove predicted noise at each step
Latent Diffusion: compress images to Latent Space with VAE then perform diffusion (64x compute reduction)
Classifier-Free Guidance: mix conditional/unconditional predictions to strengthen text conditioning
Pros
- ✓ More stable training than GANs
- ✓ Generates diverse, high-quality images
- ✓ Powerful text-conditional generation
- ✓ Also applicable to image editing/transformation
Cons
- ✗ Slow generation (requires dozens of reverse diffusion steps)
- ✗ High GPU memory requirements
- ✗ Text-image alignment may be imperfect
- ✗ Risk of generating harmful content