πŸ“‹

Evaluation Cheat Sheet by Task β€” Which Metric to Use at a Glance

Classification→Accuracy/F1, Generation→BLEU/ROUGE, OCR→CER/WER, Separation→SDR — with code

All via pip install evaluate. 3 lines of code.

Cheat Sheet

Task Primary Good Level Key
Classification Accuracy, F1 85%+ accuracy, f1
Translation BLEU 0.3+ bleu
Summarization ROUGE-L 0.4+ rouge
OCR CER <3% cer
STT WER <10% wer
Source separation SDR 10+ dB mir_eval
Text generation BERTScore 0.85+ bertscore

Key Concepts

1

Classification β†’ load("accuracy"), load("f1")

2

Translation/Summary β†’ load("bleu"), load("rouge")

3

OCR/STT β†’ load("cer"), load("wer")

4

Semantic similarity β†’ load("bertscore")

5

Source separation β†’ mir_eval.separation.bss_eval_sources()

Use Cases

Fine-tuning monitoring β€” auto-evaluate per epoch with Trainer compute_metrics Model selection β€” compare CER of TrOCR vs EasyOCR vs PaddleOCR on same data Production criteria β€” quality gates like "deploy if CER <3%, BLEU >0.4"

References

Related

πŸ“Š
AI Model Evaluation Guide β€” How Do You Know Your Model Is Good?
Accuracy, BLEU, CER, SDR β€” different metrics per task and why each is used
→
← πŸ“Š AI Model Evaluation Guide β€” How Do You Know Your Model Is Good? πŸ§ͺ AI Evaluation Tools Catalog β€” evaluate, lm-eval-harness, RAGAS, DeepEval →