The open-source evaluation framework for RAG systems and AI Agents.
Local-first. Framework-agnostic. Developer-friendly.
Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings pytest-level simplicity to AI evaluation — run from your terminal, get actionable insights, and ship with confidence.
- Local-first — No cloud services, dashboards, or authentication required
- Framework-agnostic — Works with LangChain, LlamaIndex, or any custom RAG pipeline
- 18+ metrics — Retrieval, generation, faithfulness, relevancy, performance, and cost
- Plugin-based — Extend with custom metrics, providers, and report generators
- Production-ready — Corpus auditing, failure diagnosis, and synthetic test data generation
pip install openagent-evalFor development:
git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval
uv syncoaeval init --interactive
oaeval validate config.yamloaeval run config.yamloaeval report latestOpenAgent Eval is community-built and welcomes contributors of every experience level. If you've used the tool, you already know something worth sharing — a bug, a rough edge in the docs, or a metric you wish existed.
Ways to help:
- Code — pick up a good first issue or something labeled help wanted, fix a bug, or add a metric or provider.
- Docs — clarify a guide, fix a typo, or expand the documentation.
- Bug reports — hit something unexpected? Open an issue with steps to reproduce.
- Tutorials & examples — show how you use OpenAgent Eval by adding to
examples/.
Getting started:
- Fork the repository and create a feature branch (
git checkout -b feature/your-change) - Make your change and add tests where it makes sense
- Run the checks (
uv run pytestanduv run ruff check .) - Open a pull request
New to the project? CONTRIBUTING.md walks through setup, the development workflow, and coding conventions in detail.
Where to connect:
- GitHub Discussions — ask questions, share ideas, and show what you built
- GitHub Issues — bug reports and feature requests
- Documentation — guides and API reference
- Changelog — release history
| Feature | Description |
|---|---|
| CLI + SDK | Use via command line or import as a Python library |
| Beautiful Reports | Terminal, Markdown, HTML, and JSON output formats |
| Failure Analysis | Identify why evaluations fail, not just that they failed |
| Corpus Health Auditor | Detect contradictions, staleness, and duplicates before evaluation |
| LLM-as-Judge Metrics | NLI-based scoring for faithfulness and relevancy |
| Component Diagnosis | Blame attribution — retrieval vs generation vs chunking |
| Synthetic Test Data | Auto-generate test cases from your knowledge base |
Retrieval Metrics
- Context Precision & Recall
- Precision@K & Recall@K
- Hit Rate
- Mean Reciprocal Rank (MRR)
- Normalized Discounted Cumulative Gain (NDCG)
Generation Metrics
- Faithfulness (NLI-based)
- Answer Relevancy (NLI-based)
- Hallucination Detection
- Semantic Similarity
- Exact Match & F1 Score
- BLEU & ROUGE
- BERTScore
- LLM-as-Judge (custom criteria)
Performance & Cost
- Latency tracking (embedding, retrieval, LLM stages)
- Token counting (prompt, completion, total)
- Cost estimation per provider
LLM Providers: OpenAI · Anthropic · Google Gemini · Groq · OpenRouter · Ollama
Retriever Providers: Chroma · Qdrant · Pinecone · Weaviate · FAISS · pgvector · Elasticsearch · BM25
| Command | Description |
|---|---|
oaeval init |
Create configuration file (interactive wizard) |
oaeval run <config> |
Run evaluation pipeline |
oaeval report <id> |
View evaluation reports |
oaeval compare <a> <b> |
Compare two experiments |
oaeval list |
List previous evaluations |
oaeval validate <config> |
Validate configuration |
oaeval doctor |
Check environment and dependencies |
oaeval audit --corpus <path> |
Audit corpus health |
oaeval diagnose --report <id> |
Diagnose failures and attribute blame |
oaeval synth --corpus <path> |
Generate synthetic test cases |
from openagent_eval.core import Engine
from openagent_eval.config import load_config
config = load_config("config.yaml")
engine = Engine(config)
report = await engine.run(dataset)
print(report.summary)openagent-eval/
├── openagent_eval/
│ ├── cli/ # CLI commands (Typer)
│ ├── config/ # Configuration system (Pydantic)
│ ├── core/ # Core orchestration engine
│ ├── metrics/ # 18+ evaluation metrics
│ ├── providers/ # LLM & Retriever adapters
│ ├── corpus/ # Corpus Health Auditor
│ ├── diagnosis/ # Component Diagnosis
│ ├── synthesis/ # Synthetic Test Data
│ ├── reports/ # Report generators
│ └── plugins/ # Plugin system
├── tests/ # Test suite
├── docs/ # Documentation
└── examples/ # Tutorials and examples
Licensed under the Apache License, Version 2.0. See LICENSE for details.
Built with care by the OpenAgentHQ community.