How to Build a Zero-Shot Voice Cloning TTS Service in 3 Steps with OmniVoice

Have you ever run into scenarios like wanting to dub a foreign language video but finding translation + voiceover costs too high, or wanting to produce an audiobook without access to a suitable narrator? Maybe you just want to record a short clip of your voice and have AI read your lengthy technical documents for you.

In the past, achieving this meant either paying premium rates for professional voice actors or wrestling with complex TTS toolchains that were costly to train and yielded inconsistent results.

Today, I'll walk you through OmniVoice, a high-quality voice cloning TTS project supporting 600+ languages. We'll build a local service from scratch where you simply input a reference audio clip and text, and it generates speech in that exact voice. Once set up, you can use it for video dubbing, multilingual content creation, or even generating audio versions of your articles.

Follow along, and you'll see results in about 20 minutes. Let's dive in.

Prerequisites

Before we start, ensure your environment meets these basic requirements:

• Python 3.8+: Highly recommended to use conda or venv for an isolated environment to avoid package conflicts.
• NVIDIA GPU (Recommended): 8GB+ VRAM is sufficient for smooth operation. Apple Silicon (M1/M2/M3) and Intel Arc GPUs are also supported, though performance may vary.
• Basic Python Knowledge: Familiarity with pip and reading simple Python scripts is enough.

Why recommend a GPU? OmniVoice is built on a diffusion-based TTS architecture. Inference on a GPU is up to 40x faster (Real-Time Factor as low as 0.025). CPU will work, but generating a single sentence might take tens of seconds.

Step 1: Install OmniVoice

The project offers two installation methods. Using pip is the most straightforward.

1. Install PyTorch First

Choose the version that matches your hardware. For NVIDIA with CUDA 12.8:

pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128

For Apple Silicon users:

pip install torch==2.8.0 torchaudio==2.8.0

Why two steps? OmniVoice relies on PyTorch's GPU acceleration capabilities. PyTorch has numerous CUDA builds tightly coupled with system GPU drivers. Installing separately prevents version conflicts and makes it easier to update if you switch hardware later.

2. Install OmniVoice

The stable version from PyPI is usually sufficient:

pip install omnivoice

If you want the latest features, install directly from GitHub:

pip install git+https://github.com/k2-fsa/OmniVoice.git

Verify installation: Run omnivoice-demo --help in your terminal. If you see the help menu, you're good to go.

Step 2: Run Your First Voice Clone

Instead of jumping straight into code, let's experience the core feature first: zero-shot voice cloning.

What is Zero-Shot Voice Cloning?

Simply put: you provide the model with a 3–10 second reference audio clip and tell it what the audio says. The model learns that voice profile and uses it to read any new text you provide. No training, no fine-tuning—it works instantly during inference.

Python API in Action

Create a clone_voice.py file in your project directory and paste the following:

from omnivoice import OmniVoice
import soundfile as sf
import torch

## Load the pre-trained model onto GPU
model = OmniVoice.from_pretrained(
    "k2-fsa/OmniVoice",
    device_map="cuda:0",
    dtype=torch.float16
)

## Apple Silicon users: use device_map="mps"
## Intel Arc users: use device_map="xpu"

## Voice cloning: provide reference audio and its corresponding text
audio = model.generate(
    text="Today, let's talk about the latest advancements in large language models. It's a fascinating topic.",
    ref_audio="ref.wav",
    ref_text="This is a sample recording used for voice cloning.",
)

## audio is a list of numpy arrays with a 24kHz sample rate
## The output is a 24kHz waveform, saved directly as a wav file
sf.write("output_cloned.wav", audio[0], 24000)
print("Voice cloning complete. Saved to output_cloned.wav")

Before running, prepare a ref.wav file—a clean 3–10 second voice recording. If you don't have text for it, that's fine. You can omit ref_text, and the model will automatically transcribe it using Whisper ASR:

## Omit ref_text, model will auto-transcribe
audio = model.generate(
    text="Today, let's talk about the latest advancements in large language models. It's a fascinating topic.",
    ref_audio="ref.wav",
)

Why can you skip ref_text? The model has a built-in Whisper ASR module that automatically recognizes the reference audio content. This is especially user-friendly for non-English languages—you just drop the audio file without manual transcription.

Run python clone_voice.py. After a short wait, output_cloned.wav will appear. Open it with your system player to hear the result.

Step 3: Voice Design—Customize Voices Without Reference Audio

Sometimes you don't want to clone a real person's voice, but rather "design" one from scratch. OmniVoice's Voice Design mode lets you describe the desired voice attributes using natural language.

audio = model.generate(
    text="Hello, this is a test of zero-shot voice design.",
    instruct="female, low pitch, british accent",
)
sf.write("voice_design.wav", audio[0], 24000)

Supported attributes include:

• Gender: male / female
• Age: child / young / middle-aged / elderly
• Pitch: very low / low / medium / high / very high
• Style: whisper (whispering mode)
• English Accents: American, British, etc.
• Chinese Dialects: Sichuan, Shaanxi, etc.

Feel free to mix and match. Note that Voice Design is currently primarily trained on English and Chinese data. Other languages may work but with varying stability.

Practical Project: Batch Generate Voiceover Files

Running a single example is great, but let's tackle a real-world scenario: you have 10 English technical documents and want to dub them with a consistent custom voice.

Step 1: Clone Your Voice & Save the Prompt

## Clone and save the voice profile
prompt = model.create_voice_clone_prompt(
    ref_audio="my_voice.wav",
    ref_text="This is a sample of my voice."
)
prompt.save("my_voice.pt")

## Load directly next time, skipping re-cloning
from omnivoice import VoiceClonePrompt
prompt = VoiceClonePrompt.load("my_voice.pt")

Why save the prompt? Encoding the reference audio takes time every time you clone. Saving it as a .pt file allows direct loading for significantly faster inference later.

Step 2: Batch Generation

texts = [
    "Large language models have revolutionized natural language processing.",
    "Fine-tuning allows models to adapt to specific domains efficiently.",
    "Retrieval-augmented generation reduces hallucination in AI systems."
]

for i, text in enumerate(texts):
    audio = model.generate(
        text=text,
        voice_clone_prompt=prompt,
        speed=1.0,      # Speech rate: >1 faster, <1 slower
        num_step=32,    # Diffusion steps: 16 is faster but slightly lower quality
    )
    sf.write(f"output_{i:03d}.wav", audio[0], 24000)
    print(f"Generated: output_{i:03d}.wav")

Once finished, you'll have a complete sequence of voiceover files ready to drag into CapCut or Premiere for video editing.

Common Issues & Pro Tips

Q1: Model download is too slow or times out?
Common for users in certain regions. Set the Hugging Face mirror before running:

export HF_ENDPOINT="https://hf-mirror.com"

Q2: Numbers in Chinese are read digit-by-digit?
E.g., "我有 2345 个苹果" reads as "two three four five" instead of "two thousand three hundred forty-five". Enable text normalization:

audio = model.generate(
    text="我有 2345 个苹果",
    ref_audio="ref.wav",
    normalize_text=True
)

Requires extra dependency: pip install "omnivoice[tn]". If installation fails on macOS, run conda install -c conda-forge pynini first.

Q3: Want faster inference?
If you have an NVIDIA GPU, install FlashInfer to boost throughput by 2x–2.6x:

pip install flashinfer-python==0.6.15.post1 \
    --extra-index-url https://flashinfer.ai/whl/cu128/

Q4: Poor cross-lingual cloning results?
Cloning works best when the reference audio and target text share the same language. If your reference is Chinese and you generate English, the output will carry a Chinese accent. This is expected behavior, as the model inherits pronunciation habits from the reference audio.

Summary

Today we covered three key milestones:

1. Environment Setup: Installing PyTorch and OmniVoice via pip, understanding the two-step install rationale.
2. Voice Cloning Workflow: From zero-shot cloning to saving voice prompts and batch generation.
3. Voice Design & Optimization: Customizing voices with natural language, fixing number normalization, and enabling FlashInfer acceleration.

You now have a local TTS service supporting 600+ languages. Your next steps could include:

• Running omnivoice-demo to launch a Web UI for drag-and-drop testing.
• Wrapping this capability into an HTTP service using FastAPI to integrate into your content pipeline.
• Exploring non-verbal token controls, like inserting [laughter] for more natural AI expressions.

The tools are in your hands. Go build your first AI voiceover project, and reach out if you hit any snags.