Vision-Language Model
AI that understands both images and text together
Vision-Language Model (VLM) is AI that comprehensively understands visual information and text. Images are converted to embedding vectors via a Vision Encoder (typically ViT โ Vision Transformer), then processed alongside LLM text tokens. A Projection Layer (linear transformation or Q-Former) aligns visual embeddings to the LLM's text space. Representatives include GPT-4V, Claude 3 Vision, Gemini, and LLaVA. Various visual tasks are now possible: OCR, chart interpretation, medical imaging analysis, and code screenshot understanding.
Key Concepts
Input image to Vision Encoder (ViT) โ generate per-patch embedding vectors
Projection Layer transforms visual embeddings to LLM's text embedding space
Feed converted image tokens + text tokens together into the LLM
LLM's Self-Attention learns relationships between image and text tokens
Cross-modal understanding: "Where is the cat in this image?" โ integrated visual+language reasoning
LLM generates text response (image description, Q&A, analysis, etc.)
Pros
- ✓ Handles combined image+text tasks
- ✓ Text recognition without separate OCR
- ✓ Understands charts/graphs/diagrams
- ✓ Foundation for multimodal Agents
Cons
- ✗ Hallucination โ generating content not in the image
- ✗ Image resolution/token count limitations
- ✗ Difficulty recognizing subtle visual differences
- ✗ Visual biases in training data