Transformer
Parallel processing architecture based on Self-Attention
Transformer is an architecture proposed in Google's 2017 paper "Attention Is All You Need." Previous RNN/LSTM models processed sequences sequentially, making parallelization difficult and losing information in long sentences. Transformer solved this by using Self-Attention to reference all positions in a sequence simultaneously. It computes attention scores between tokens using Query, Key, and Value vectors, with Positional Encoding to supplement position information. It is the foundation of modern LLMs including GPT (decoder-only), BERT (encoder-only), and T5 (encoder+decoder).
Key Concepts
Split input text into tokens β convert each token to an embedding vector
Add Positional Encoding β add position information to vectors
Generate Query (Q), Key (K), Value (V) vectors for each token
Compute Attention Score: softmax(QK^T / βd) Γ V β determines how much each token attends to others
Multi-Head Attention: perform multiple attentions in parallel β capture diverse relationship patterns
Generate final output with Feed-Forward Network + Layer Normalization + Residual Connection
Pros
- ✓ Fully parallelizable β maximizes GPU utilization
- ✓ No information loss in long sequences (Self-Attention)
- ✓ Universally applicable to diverse tasks
- ✓ Scaling law (larger models yield better performance)
Cons
- ✗ O(nΒ²) memory β memory usage scales quadratically with sequence length
- ✗ Requires massive GPU/TPU for large-scale training
- ✗ Position information must be encoded separately
- ✗ Directly reflects biases in training data