π
AI Model Evaluation Guide β How Do You Know Your Model Is Good?
Accuracy, BLEU, CER, SDR β different metrics per task and why each is used
Why Evaluation Is Needed
You ran trainer.train(). But how do you know your model is actually good? "Looks OK when I try it" is not evaluation. You need numbers.
Core Concept: Test Data with Known Answers
Training data: model learns from (study materials)
Test data: model is evaluated on (exam questions)
Using study materials as the exam is meaningless. Evaluate on data the model has never seen.
Metrics by Task
| Task | Metric | Direction | Code |
|---|---|---|---|
| Classification | Accuracy, F1 | Higher β | load("accuracy") |
| Translation | BLEU | Higher β | load("bleu") |
| Summarization | ROUGE | Higher β | load("rouge") |
| OCR | CER, WER | Lower β | load("cer") |
| Source separation | SDR | Higher β | mir_eval |
All available via pip install evaluate β load("metric_name").
Fine-tuning + Evaluation = Set
Add compute_metrics to Trainer β see accuracy per epoch during training.
Key Concepts
1
pip install evaluate β install HuggingFace evaluation library
2
metric = load("accuracy") β load metric matching your task
3
metric.compute(predictions, references) β compare predictions to answers β one number
4
Add compute_metrics to Trainer β auto-evaluate every epoch during training
Use Cases
Fine-tuning quality check β track accuracy/CER/BLEU per epoch to prevent overfitting
Model comparison β compare multiple models on same test data
Deployment decision β set criteria like "deploy if CER below 3%"