Speech To Speech: Build voice agents with open-source models

A low-latency, fully modular voice-agent pipeline: VAD -> STT -> LLM -> TTS, exposed through an OpenAI Realtime-compatible WebSocket API. Every component is swappable. The LLM slot speaks OpenAI-compatible protocols, so you can point it at a hosted provider, at HF Inference Providers, or at a vLLM or llama.cpp server on your own hardware for a fully local, fully open stack.

This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots.

Quickstart

pip install speech-to-speech
export OPENAI_API_KEY=...
speech-to-speech

This starts an OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime using Parakeet TDT for local STT, an OpenAI-compatible LLM, and Qwen3-TTS for local speech output.

From a source checkout, talk to it from a second terminal:

python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765

Prefer to keep the LLM on your own machine? Serve Gemma 4 with llama.cpp:

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

Then point the OpenAI-compatible LLM backend at it:

speech-to-speech \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

Any OpenAI Realtime-compatible client can connect. See Realtime API for the protocol and LLM backends for provider and local-server options.

Index

• How it works
• Installation
• Supported components
• Run modes
• Realtime API
• LLM backends
• Multi-language support
• Pocket TTS
• CLI reference
• Contributing
• Citations

How it works

The pipeline is a cascade of four components, each running in its own thread and connected by queues:

1. Voice Activity Detection (VAD): Silero VAD v5 detects speech boundaries and turn-taking.
2. Speech to Text (STT): transcribes the user's turn, with optional live partial transcripts.
3. Language Model (LLM): generates the response, streaming text and tool calls.
4. Text to Speech (TTS): synthesizes audio and streams it back to the client.

Every stage has multiple interchangeable backends, selected via CLI flags. The code is designed for easy modification, with a focus on models available through Transformers and the Hugging Face Hub.

Installation

Requires Python 3.10+.

pip install speech-to-speech

The default install covers the standard realtime path:

• Parakeet TDT for STT
• OpenAI-compatible API for the language model
• Qwen3-TTS for speech output, using the GGML backend by default on non-macOS platforms and mlx-audio on Apple Silicon
• local audio and realtime server modes

macOS and non-macOS dependencies are resolved automatically via platform markers in pyproject.toml.

CUDA Note for Qwen3-TTS

On Linux, the Qwen3-TTS GGML backend comes from faster-qwen3-tts[ggml]. Its default qwentts-cpp-python wheel on PyPI targets CUDA 12.8. If your machine does not have the CUDA 12 runtime that wheel expects, install the matching wheel from the Hugging Face wheelhouse before installing speech-to-speech:

# CUDA 13.x
pip install "qwentts-cpp-python==0.3.0+cu130" \
  -f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu130

# CUDA 12.4
pip install "qwentts-cpp-python==0.3.0+cu124" \
  -f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cu124

# CPU-only fallback
pip install "qwentts-cpp-python==0.3.0+cpu" \
  -f https://huggingface.co/datasets/andito/qwentts-cpp-python-wheels/tree/main/whl/cpu

pip install speech-to-speech

To use the previous CUDA-graphs implementation instead of GGML, pass --qwen3_tts_backend torch.

Optional Backends

Extra backends are installed with pip extras:

pip install "speech-to-speech[kokoro]"          # Kokoro-82M TTS on non-macOS
pip install "speech-to-speech[pocket]"          # Pocket TTS
pip install "speech-to-speech[chattts]"         # ChatTTS
pip install "speech-to-speech[facebook-mms]"    # MMS TTS
pip install "speech-to-speech[faster-whisper]"  # Faster Whisper STT
pip install "speech-to-speech[whisper-mlx]"     # Lightning Whisper MLX STT on macOS
pip install "speech-to-speech[paraformer]"      # Paraformer STT through FunASR
pip install "speech-to-speech[mlx-lm]"          # mlx-vlm support for vision models on macOS

Deprecated implementations, including MeloTTS, live in archive/ and are no longer wired into the CLI.

Note on DeepFilterNet: DeepFilterNet, used for optional audio enhancement in VAD, requires numpy<2 and conflicts with Pocket TTS, which requires numpy>=2. Install it manually only in environments where you are not using Pocket TTS.

From Source

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech
uv sync

This installs the package in editable mode and makes the speech-to-speech CLI available.

Supported Components

Select implementations with --stt, --llm_backend, and --tts. Run speech-to-speech -h for exact values and backend-specific flags.

Run Modes

Realtime Server

export OPENAI_API_KEY=...
speech-to-speech

This is equivalent to:

speech-to-speech \
    --thresh 0.6 \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --tts qwen3 \
    --qwen3_tts_model_name Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --qwen3_tts_speaker Aiden \
    --qwen3_tts_language auto \
    --qwen3_tts_backend ggml \
    --qwen3_tts_non_streaming_mode True \
    --qwen3_tts_mlx_quantization 6bit \
    --model_name gpt-5.4-mini \
    --chat_size 30 \
    --responses_api_stream \
    --enable_live_transcription \
    --mode realtime

The default model is gpt-5.4-mini through the OpenAI Responses API. Override it with --model_name, and set --responses_api_base_url for another OpenAI-compatible provider or server.

Local Mac

speech-to-speech --local_mac_optimal_settings

Optionally with a specific LLM:

speech-to-speech \
    --local_mac_optimal_settings \
    --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16

This setting:

• Adds --device mps to use MPS for all models.
• Sets Parakeet TDT for STT.
• Sets MLX LM as the LLM backend.
• Sets Qwen3-TTS for TTS, using mlx-audio with the 6bit MLX variant by default.
• Sets --mode local.

--tts pocket and --tts kokoro are also valid on macOS.

To compare the MLX quantization variants locally:

python scripts/benchmark_tts.py \
    --handlers qwen3 \
    --iterations 3 \
    --qwen3_mlx_quantizations bf16 4bit 6bit 8bit

WebSocket

1. Run the pipeline in WebSocket mode:

   speech-to-speech --mode websocket --ws_host 0.0.0.0 --ws_port 8765

2. Connect from your client at ws://<server-ip>:8765. Send raw audio bytes as 16 kHz, int16, mono PCM and receive generated audio bytes back.

TCP Socket

TCP socket mode is intentionally minimal. It streams raw PCM audio, but does not provide the full Realtime API feature set, including interruption handling, live transcript events, or tool-call events.

1. Run the pipeline on the server:

   speech-to-speech --mode socket --recv_host 0.0.0.0 --send_host 0.0.0.0

2. Run the client locally to handle microphone input and playback:

   python scripts/listen_and_play.py --host <IP address of your server>

Docker

Install the NVIDIA Container Toolkit, then:

docker compose up

The compose file starts a llama.cpp server with Gemma 4, starts the TCP socket server, and exposes ports 8080, 12345, and 12346.

Realtime API

Realtime mode streams audio over a WebSocket using the OpenAI Realtime protocol, with live transcription and low-latency turn-taking. The server exposes /v1/realtime, and any OpenAI Realtime-compatible client can connect:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    conn.send(
        {
            "type": "session.update",
            "session": {
                "type": "realtime",
                "instructions": "You are a helpful assistant.",
                "audio": {
                    "input": {
                        "turn_detection": {
                            "type": "server_vad",
                            "interrupt_response": True,
                        }
                    }
                },
            },
        }
    )

    for event in conn:
        print(event.type)

The server implements the core Realtime event set: input_audio_buffer.append, session.update, conversation.item.create, response.create, and response.cancel inbound; speech start/stop, streaming transcription, audio deltas, tool calls, and response.done outbound. The full event reference, architecture, and design details live in the Realtime Engine README.

LLM Backends

The LLM is the most compute-intensive and highest-latency component in the pipeline. A single forward pass through a large model can dominate end-to-end response time, so choosing the right backend for your hardware and latency budget matters. The pipeline supports:

• Local inference: transformers on CUDA / CPU and mlx-lm on Apple Silicon.
• Self-hosted servers: responses-api and chat-completions can point at a local vLLM or llama.cpp server.
• Provider APIs: the same backends work with OpenAI, HF Inference Providers, OpenRouter, and other OpenAI-compatible providers.

Two API backends are available, sharing the same --responses_api_* connection flags:

• --llm_backend responses-api (default) targets /v1/responses.
• --llm_backend chat-completions targets /v1/chat/completions.

The examples below pair Parakeet TDT for local STT and Qwen3-TTS for local TTS with different LLM backends.

Responses API Backend

Works with any provider or server that implements the OpenAI Responses API. Point --responses_api_base_url at the endpoint and set --model_name accordingly:

# OpenAI
speech-to-speech \
    --mode local \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --tts qwen3 \
    --qwen3_tts_mlx_quantization 6bit \
    --model_name "gpt-4o-mini" \
    --responses_api_api_key "$OPENAI_API_KEY" \
    --responses_api_stream \
    --enable_live_transcription

# HF Inference Providers: Qwen3.5-9B via Together
speech-to-speech \
    --mode local \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --tts qwen3 \