csm.rs is a high-performance Rust implementation of Sesame's Conversational Speech Model (CSM), designed for fast, efficient, and real-time streaming text-to-speech (TTS) inference. It is built on the candle machine learning framework.

This implementation is simple, straightforward, and aims for raw performance.

• ⚡️ Blazing-Fast: High-performance inference powered by Rust and candle.
• 🤗 Broad Model Support: Natively supports both the original sesame/csm-1b weights and weights from Hugging Face transformers-compatible fine-tunes.
• 🤏 Quantization: Supports GGUF-based q8_0 and q4_k quantization for reduced memory footprint and faster inference on CPU.
• ⚙️ Multiple Backends: Leverages candle to support multiple hardware targets, including MKL, Accelerate (macOS), CUDA, cuDNN, and Metal (Apple Silicon).
• 🔌 OpenAI Compatible: Includes an OpenAI-compatible API web server for seamless integration with existing tools.

To build the project, select the appropriate feature flag for your target hardware. The project provides three main binaries: main (for command-line interface usage), benchmark (for throughput measurement), and server (for the OpenAI-compatible API).

CPU (MKL - Linux/Windows) For optimal performance on Intel CPUs.

RUSTFLAGS="-C target-cpu=native" cargo build --release --features mkl

CPU (Accelerate - macOS) For optimal performance on Apple CPUs.

RUSTFLAGS="-C target-cpu=native" cargo build --release --features accelerate

NVIDIA GPU (CUDA) Requires the CUDA Toolkit to be installed.

cargo build --release --features cuda

NVIDIA GPU (cuDNN) For faster CUDA performance with cuDNN.

cargo build --release --features cudnn

Apple Silicon GPU (Metal) For running on M-series Macs.

cargo build --release --features metal

The compiled binaries will be available in the ./target/release/ directory.

The CLI allows you to generate audio directly from your terminal. Models are downloaded automatically from the Hugging Face Hub on first use.

Generate audio with a full-precision model:

./target/release/main \
    --text "Hello there from the full precision model" \
    --model-id "sesame/csm-1b" \
    --output "output_fp16.wav"

Generate audio with a quantized model:

./target/release/main \
    --text "Hello there from the quantized model" \
    --model-id cartesia/sesame-csm-1b-gguf \
    --model-file q8.gguf \
    --output "output_q8.wav"

To quantize your own models see the Quantization section.

csm.rs includes a server that is compatible with the OpenAI Speech API, allowing you to use it as a drop-in replacement.

Start the server with a full-precision model:

./target/release/server --port 8080 --model-id "sesame/csm-1b"

Start the server with a quantized model:

./target/release/server \
    --port 8080 \
    --model-id cartesia/sesame-csm-1b-gguf \
    --model-file q8.gguf

Python Client Example You can use the official OpenAI Python client to interact with the server.

# pip install openai
from openai import OpenAI
from pathlib import Path

# Point the client to your local server
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

# Request speech synthesis
response = client.audio.speech.create(
    model="csm-1b", # Model name is ignored by the server but required by the API
    input="Hello! This audio was generated by the server.",
    voice="alloy", # Voice is ignored, use speaker_id instead
    # You can pass custom parameters in extra_body
    extra_body={
        "speaker_id": 0,
        "temperature": 0.7,
    }
)

# Save the output to a file
speech_file_path = Path("server_output.wav")
response.stream_to_file(speech_file_path)

# Or use the streaming endpoint
with client.audio.speech.with_streaming_response.create(
    model="csm-1b",
    voice="alloy",
    input="Hello from the streaming endpoint",
    response_format="wav",
    extra_body=dict(
        speaker_id=0,
    )
) as response:
    for chunk in response.iter_bytes(chunk_size=1024):
        print(chunk)

All binaries share a common set of arguments for model loading and hardware selection.

You can significantly reduce the model size and improve CPU inference speed by quantizing the weights to 8-bit (q8_0) or 4-bit (q4_k). We use the GGUF file format for quantized models.

A Python script is provided to handle downloading, loading, and converting .safetensors weights into a quantized GGUF file. The script can work directly with both single-file and sharded models from local paths or the Hugging Face Hub.

1. Install dependencies:

   pip install -r scripts/requirements.txt

2. Run the quantization script:

   The script can quantize a model directly from the Hugging Face Hub or from a local directory.

   To quantize a model from the Hub (e.g., sesame/csm-1b) to Q8_0:

   python scripts/quantize.py \
       --model-id "sesame/csm-1b" \
       --index-file "transformers.safetensors.index.json" \
       --output-path ./csm-1b-q8_0.gguf \
       --qtype q8_0

   To quantize a local model to Q4_K:

   python scripts/quantize.py \
       --model-path /path/to/your/local/model/directory \
       --output-path ./csm-1b-q4_k.gguf \
       --qtype q4_k

You can run the built-in benchmark tool to measure the performance on your hardware. The tool reports the Real-Time Factor (RTF), which is the time taken to generate 1 second of audio (lower is better), and Throughput (xRealTime), which is how many seconds of audio are generated in 1 second (higher is better).

Example benchmark command:

# For a full-precision model with CUDA
cargo run --release --features cuda --bin benchmark

# For a quantized model on CPU
RUSTFLAGS="-C target-cpu=native" cargo run --release --features mkl --bin benchmark -- --weights-path ./csm-1b-q8_0.gguf

This project is licensed under the GNU Affero General Public License Version 3. See the LICENSE file for details.

Contributions are welcome!

If you have suggestions for improvements, find a bug, or want to add a new feature, please feel free to open an issue or submit a pull request.