Pre-training
The stage of learning general knowledge from massive data
Pre-training is the first stage of LLM training, performing self-supervised learning on trillions of tokens (books, web pages, code, etc.). GPT models use "next token prediction" (Causal Language Modeling), while BERT uses "masked token prediction" (Masked Language Modeling). Through this process, general capabilities in grammar, common sense, reasoning, coding, and math naturally emerge. It takes weeks to months on thousands of GPUs, with costs reaching tens of millions of dollars. Pre-training alone doesn't enable instruction following or conversation โ Fine-tuning and RLHF are needed afterward.
Key Concepts
Collect large-scale text corpus (web crawling, books, code, papers, etc.)
Data preprocessing โ deduplication, harmful content filtering, tokenization
Initialize model architecture (Transformer-based, random weights)
Repeat next token prediction training โ "The cat sat on the [?]" โ predict "mat"
Training on thousands of GPUs for weeks to months (distributed training)
Base Model complete โ can complete text but cannot yet follow instructions or converse
Pros
- ✓ Can learn from unsupervised (unlabeled) data alone
- ✓ Forms general knowledge and reasoning capabilities
- ✓ Applicable to various downstream tasks
- ✓ Emergent abilities appear with scaling
Cons
- ✗ Enormous costs (GPU, power, data)
- ✗ Absorbs biases and harmful content from training data
- ✗ Hallucination inherently built-in
- ✗ Cannot follow instructions/converse (additional training needed)