Open-source evaluation workspace for production voice AI agents.

Book a demo · Visit VaaniEval · Star on GitHub

Development | Architecture | Deployment | Contributing | Discussions

VaaniEval helps voice-agent teams inspect real conversations, run evaluator-backed quality checks, and turn production call data into actionable QA, product, and engineering feedback.

VaaniEval is a full-stack application, not a published Python package or supported CLI. It consists of a FastAPI backend, a queue worker, a React product app, and a Next.js public site.

Watch the VaaniEval product demo for a walkthrough of the conversation review and quality-evaluation workspace.

• Production conversation imports from supported voice providers
• Transcript and audio review in a purpose-built conversation workspace
• Evaluator-backed scores with rationales and conversation evidence
• Dashboard analytics for quality trends, KPIs, and agent-level drilldowns
• Encrypted provider credentials and workspace-scoped data
• Queue-backed imports and evaluations

Use VaaniEval to examine task completion, resolution quality, unsupported claims, fallback behavior, and available operational signals.

• Python 3.11+
• Node.js 20+ and npm
• Git
• ElevenLabs, Vapi, or Bolna credentials for importing conversations
• Evaluator provider credentials, such as OpenAI, for scoring

Windows:

./start-dev.cmd

or:

./start-dev.ps1

macOS/Linux:

chmod +x start-dev.sh
./start-dev.sh

Services:

• Frontend: http://localhost:5173
• Backend API: http://localhost:8000
• Backend API docs: http://localhost:8000/docs
• Worker: started by the development scripts

1. Open the frontend.
2. Sign in with the local development flow.
3. Connect a voice provider in Provider settings.
4. Import conversations.
5. Open the Conversations workspace.
6. Trigger an evaluation and inspect scores, rationales, transcript, and audio.

For manual setup, configuration, tests, and troubleshooting, see Development.

Provider support is adapter-based. New voice providers should live behind backend provider adapters so provider-specific behavior stays isolated.

.
|-- backend/       # FastAPI API, worker, migrations, and provider adapters
|-- frontend/      # Authenticated React product
|-- site/          # Public Next.js acquisition site
|-- datasets/      # Historical evaluation datasets
|-- docs/          # Development and architecture guides
|-- tests/         # Backend, provider, and evaluation tests
`-- start-dev.*    # Local full-stack launchers

• Development: local setup, configuration, tests, and troubleshooting
• Architecture: components, data flow, queue, and security boundaries
• Deployment: production topology and deployment commands
• Contributing: contribution workflow and required checks

VaaniEval is designed for production conversation review, so treat credentials and call data carefully:

• Never commit real provider API keys or evaluator tokens.
• Keep local secrets in .env files that are not tracked.
• Use the backend Provider settings flow for connected provider credentials where possible.
• Configure production database, cookie, SMTP, CORS, and encryption secrets before deployment.

If a credential is exposed, rotate it immediately.

Contributions are welcome. Before opening a pull request, run the relevant checks and read CONTRIBUTING.md.

Need help setting up or deploying VaaniEval? Email shubham@vaanieval.com. I am happy to help you get it running.

Join GitHub Discussions to ask setup and usage questions, compare voice-agent evaluation approaches, propose ideas, or share what you have built. Use GitHub Issues for reproducible defects and scoped implementation work.

See the community support guide for category guidance and safe-posting expectations.

VaaniEval is open-source software licensed under the GNU Affero General Public License v3.0 (AGPL-3.0-only). You may use, modify, and distribute it under that license. If you modify and offer the software to users over a network, AGPL section 13 requires you to offer those users the Corresponding Source of your modified version.