You have 60 lines of narration to record, a client who wants the files by Thursday, and a budget that covers exactly zero studio hours. That was the week Coqui AI stopped being an interesting GitHub repository for me and started being the thing that actually shipped the job.
What follows is the process I wish someone had handed me: environment, reference clip, first clone, then a batch run of a few hundred lines. Every command below is one I have run. Nothing here is theoretical.
What Coqui AI Actually Does in This Workflow
Coqui AI is an open-source text-to-speech toolkit, and the part that matters for cloning is XTTS v2. It takes a short reference recording, plus a line of text, and returns that text in something that sounds like the person on the recording. The same checkpoint handles 17 languages, which is why it turns up so often in dubbing and localisation work.
One piece of housekeeping before you commit a weekend: the company behind Coqui wound down in early 2024. The repository and the model weights survived through community forks, and there’s a solid breakdown of the Coqui TTS toolkit and how its models are structured if you want that history. Your download itself will come from Hugging Face’s model hub, so budget a few minutes for that first pull.
Step 1: Set Up the Environment Without Losing an Evening
XTTS v2 wants Python 3.9 through 3.11. If your system interpreter is 3.12 or newer you will meet dependency errors that look sinister and are not. Pin the version up front.
python3.11 -m venv coqui
source coqui/bin/activate
pip install TTS
That pulls in PyTorch, so expect a couple of gigabytes. No GPU is required. On CPU, a ten-second clip takes somewhere between 20 and 40 seconds to render. On a mid-range card it runs close to real time, and that difference matters once you’re generating 200 files.
Step 2: Record a Reference Clip That Doesn’t Sabotage You
The biggest quality lever here isn’t a setting. It’s the audio you feed in. XTTS copies pacing, breathiness and room character as readily as it copies timbre, so a hissy sample produces hissy speech.
- Six to fifteen seconds. Anything under five and the model is guessing at your vowel space. Past twenty it starts averaging in ways that sound slightly off.
- One speaker, no overlap. A clip with a second voice in the background will bleed that voice into your output.
- Mono WAV at 22050 Hz.
ffmpeg -i take.m4a -ac 1 -ar 22050 reference.wavhandles it in one line. - Match the delivery you want back. If the finished script is calm and measured, do not read the sample like a trailer voiceover.
- A quiet room beats an expensive mic. A phone in a carpeted bedroom outperforms a condenser sitting next to a laptop fan.
Step 3: Your First Clone From the Command Line
Generate one file before you automate anything, so you can hear exactly what the model is doing with your sample.
tts --text "The harbour closes at dusk, and the last ferry leaves at nine." \
--model_name tts_models/multilingual/multi-dataset/xtts_v2 \
--speaker_wav reference.wav \
--language_idx en \
--out_path line001.wav
The first run downloads roughly 1.8 GB of weights, and every run after that starts fast. Run tts --model_name tts_models/multilingual/multi-dataset/xtts_v2 --list_speaker_idxs to see the built-in voices. They’re useful as a control: render the same sentence with a stock speaker and with your reference clip, and you’ll hear immediately whether your recording is the weak link.
Step 4: Batch 200 Lines Without Babysitting It
For real scripts the command line gets tedious, so switch to the Python API and keep the model in memory. Reloading the checkpoint for every line was costing me about ten seconds of pure overhead per file.
from TTS.api import TTS
import csv
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda")
with open("script.csv") as f:
for i, row in enumerate(csv.DictReader(f), start=1):
tts.tts_to_file(
text=row["line"],
speaker_wav="reference.wav",
language="en",
file_path=f"out/{i:03d}.wav",
)
Two habits keep batch runs clean. Split any line longer than about 250 characters at a sentence boundary before you send it, because the model drifts on paragraph-length input. And number your output files with zero padding (001, 002) so a later cat or ffmpeg concatenation assembles them in order without argument. A 200-line job that took 40 minutes line-by-line dropped to about six minutes once the model stayed loaded.
Step 5: The Post-Processing Nobody Mentions
Raw output comes out slightly hot, with a breath of room tone on the tail. Two ffmpeg passes fix most of it.
ffmpeg -i line001.wav -af "silenceremove=start_periods=1:start_threshold=-45dB,\
loudnorm=I=-16:TP=-1.5:LRA=11" line001_clean.wav
Normalise to roughly -16 LUFS for video and podcast platforms, closer to -14 for Spotify. When you stitch a full script together, leave around 350 ms of silence between clips so the joins don’t sound clipped. If you skip this stage entirely, the inconsistency between individually generated files is the first thing a listener notices.
Where the Clone Falls Short
XTTS v2 weights sit under the Coqui Public Model License, which is non-commercial. Read that line before you quote a client a price, because it decides whether this route is even available to you.
Delivery is the other boundary. Long-form narration, audiobooks and explainer voiceover are where a cloned voice holds up across minutes of audio. Ad reads, character work, anything that needs a laugh or a crack in the voice, and the flatness shows. Hosted services like ElevenLabs’ voice generator still handle that emotional range better, and they return audio in under a second rather than the three to five seconds a local render costs you. The sensible setup for a lot of small studios is both: Coqui for volume work you own outright, a hosted API for the handful of lines that carry the whole piece.
The Three Failures You’ll Actually Hit
Buzzing or smearing on long sentences
Usually a length problem disguised as a model problem. Cut the input at commas and full stops into chunks under 250 characters, generate each chunk, then join them. Quality jumps immediately, and you keep one reference clip the whole way through.
An accent bleeding in from the wrong language
Set --language_idx explicitly every time. A German reference clip forced to read English will sound German, which is occasionally perfect and usually not what you wanted. The fix is a reference recording in the target language, not a different flag.
Memory climbing until the run dies at line 60
This is almost always a loop holding onto tensors it no longer needs. Wrap generation in torch.no_grad(), delete the returned audio object after writing the file, and restart the process every hundred lines from a small wrapper script. On a 6 GB card that turns a crash at minute twelve into a finished batch.