AI-powered real-time voice interaction framework for building conversational audio agents.

aivoxplay is a modular Python framework for creating real-time, voice-enabled AI applications. It supports streaming Speech-to-Text (STT), and responsive Text-to-Speech (TTS), and Speech to Speech(STS) connected via WebSockets for low-latency performance.

TTS: Only Orpheus 3B is currenly supported.

AIVoxPlay is currently intended for experimentation and rapid prototyping. It started as a side project and remains one—a way to keep up with the fast-moving voice AI ecosystem. Expect breaking changes and evolving APIs.

git clone https://github.com/yourusername/AIVoxPlay.git
cd AIVoxPlay/aivoxplay
pip install -r requirements.txt

• OpenAI Realtime API for Speech-to-Text (STT)
• Orpheus 3B endpoint for Text-to-Speech (TTS) - OpenAI-style VLLM endpoint is required.
• Python 3.8+

Set your OpenAI API key in a .env file or your environment:

OPENAI_API_KEY=sk-...

cd examples
python agent_server.py

This will start a FastAPI server on localhost:8001 exposing a /voice WebSocket endpoint.

Open examples/sts_client.html in your browser. This client:

• Streams microphone audio to the server via WebSocket.
• Receives real-time transcripts and synthesized audio responses.
• Handles barge-in (interrupting TTS with new speech).
• Shows a transcript and playback indicator.

• /voice WebSocket API:

  • Client sends audio chunks (PCM, base64) to the server.
  • Server transcribes audio using OpenAI STT, sends text to a chat function (can be any LLM/agent), then streams TTS audio back.
  • Supports "cancel_audio" for barge-in.

• External Chat Function:

  • You can plug in any function (OpenAI, VLLM, LangChain, etc.) to generate responses.
  • See examples/agent_server.py for a minimal example.

• Current: Only OpenAI Realtime STT and Orpheus 3B TTS are supported.
• Planned:
  • More TTS and STT providers.
  • Better chat orchestration (e.g., LangGraph).
  • Improved client UX and orchestration patterns.

This package is for experimentation and learning. It is not production-ready. The API and features will change rapidly as the voice AI landscape evolves.

AIVoxPlay provides two main WebSocket endpoints for real-time voice interaction:

• /audio/in/{client_id}: Send microphone audio and control messages to the server.
• /audio/out/{client_id}: Receive transcripts and synthesized audio responses from the server.

The protocol supports streaming audio, real-time transcription, TTS playback, and barge-in (interrupt) functionality.

For detailed message formats, interaction flow, and client/server examples, see examples/README.md.

PRs and issues are welcome! This project is a playground for new ideas in real-time voice AI.