๐งช
AI Evaluation Tools Catalog โ evaluate, lm-eval-harness, RAGAS, DeepEval
From single metric measurement to LLM benchmarks, RAG evaluation, and automated testing pipelines
Different Tools for Different Purposes
| Tool | Purpose | Analogy |
|---|---|---|
| evaluate | Single metric | Thermometer |
| lm-eval-harness | LLM benchmark | Full health checkup |
| RAGAS | RAG pipeline | Specialized test |
| DeepEval | AI app testing | CI/CD tests |
Selection Guide
| Situation | Tool |
|---|---|
| Fine-tuning accuracy tracking | evaluate |
| How good is my LLM? | lm-eval-harness |
| RAG pipeline quality | RAGAS |
| AI app auto-testing (CI/CD) | DeepEval |
Key Concepts
1
evaluate โ pip install evaluate. Measure one metric. Use in Trainer
2
lm-eval-harness โ pip install lm-eval. Comprehensive LLM benchmark suite
3
RAGAS โ pip install ragas. Evaluate RAG retrieval quality + answer quality together
4
DeepEval โ pip install deepeval. Auto-test AI outputs like pytest
Use Cases
Fine-tuning QA โ track accuracy/CER per epoch with evaluate
LLM leaderboard โ measure MMLU/HellaSwag scores with lm-eval-harness
RAG chatbot validation โ measure hallucination rate, retrieval accuracy, answer relevancy with RAGAS