Open-source evaluation workspace for production Voice AI agents.

Visit VaaniEval · Star on GitHub

Quickstart | Development | Architecture | Metrics | Feature Playbooks | Discussions

VaaniEval helps voice-agent teams inspect real conversations, run evaluator-backed quality checks, and turn production call data into actionable QA, product, and engineering feedback.

VaaniEval is currently a full-stack application, not a published Python package. The source of truth is the backend/ FastAPI service, frontend/ React app, and backend worker process.

Most Voice AI teams outgrow pass/fail spot checks quickly. They need a repeatable way to import conversations, review evidence, score behavior, and understand where agents are improving or failing.

VaaniEval gives you:

• Production conversation ingestion from supported voice providers
• Transcript and audio review in a purpose-built conversation workspace
• Evaluation runs with metric scores and rationales
• Dashboard analytics for quality trends, KPIs, and agent-level drilldowns
• Provider settings for voice platforms and evaluator models
• Queue-backed imports and evaluations so long-running work does not block the app

Use VaaniEval to review and score conversations for:

• Task completion and resolution quality
• Hallucination or unsupported claims
• Fallback behavior and unresolved turns
• Latency and operational quality signals
• Agent, provider, and conversation-level trends

See Metrics and Gates for the current metric model.

• Python 3.11+
• Node.js 20+ and npm
• Git
• ElevenLabs or Vapi credentials for importing conversations
• Evaluator provider credentials, such as OpenAI, for scoring

Windows:

./start-dev.cmd

or:

./start-dev.ps1

macOS/Linux:

chmod +x start-dev.sh
./start-dev.sh

Services:

• Frontend: http://localhost:5173
• Backend API: http://localhost:8000
• Backend API docs: http://localhost:8000/docs
• Worker: started by the dev scripts

1. Open the frontend.
2. Sign in with the local development flow.
3. Connect a voice provider in Provider settings.
4. Import conversations.
5. Open the Conversations workspace.
6. Trigger an evaluation and inspect scores, rationales, transcript, and audio.

For manual setup, environment variable, migrations, and troubleshooting, see the Development Guide.

1. Import production conversations from a supported voice provider.
2. Normalize available transcripts, media, and metadata for review.
3. Run evaluator-backed scores and retain rationales with each conversation.
4. Review quality trends, then open the exact calls behind a weak result.

Provider support is adapter-based. New voice providers should live behind backend provider adapters so provider-specific behavior stays isolated.

.
|-- backend/                 # FastAPI app, models, services, worker, migrations
|-- frontend/                # React + Vite application
|-- docs/                    # Architecture, setup, feature playbooks, metrics
|-- datasets/                # Historical scenario datasets and references
|-- tests/                   # Backend and evaluation tests
|-- start-dev.cmd            # Windows launcher
|-- start-dev.ps1            # Windows PowerShell launcher
`-- start-dev.sh             # macOS/Linux launcher

• Quickstart
• Development Guide
• Backend Architecture
• Backend API Reference
• Configuration
• Metrics and Gates
• Troubleshooting
• Feature Playbooks
• V2 Plan

Run backend tests from the repository root:

pytest

Build the frontend:

cd frontend
npm run build

See docs/development.md for the full manual setup.

VaaniEval is designed for production conversation review, so treat credentials and call data carefully:

• Never commit real provider API keys or evaluator tokens.
• Keep local secrets in .env files that are not tracked.
• Use the backend Provider settings flow for connected provider credentials where possible.
• Configure production database, cookie, SMTP, CORS, encryption, and cron secrets before deployment.

If a credential is exposed, rotate it immediately.

Contributions are welcome. Good first areas include:

• New provider adapters
• Evaluation metric improvements
• Conversation review UX improvements
• Import and worker reliability
• Documentation and examples

Before opening a PR, run the relevant backend and frontend checks and include screenshots for UI changes. See docs/contributing.md for the full contribution guide.

Need help setting up or deploying VaaniEval? Email shubham@vaanieval.com. I am happy to help you get it running.

Join GitHub Discussions to ask setup and usage questions, compare voice-agent evaluation approaches, propose ideas, or share what you have built. Use GitHub Issues for reproducible defects and scoped implementation work.

See the community support guide for category guidance and safe-posting expectations.

• V2 Plan Overview
• V2 Roadmap
• V2 Score Taxonomy
• V2 Audio Scalability Plan

VaaniEval is licensed under the MIT License.