Voice cloning / voice conversion

Self-host FreeVC: one-shot voice cloning in 1.4GB of weights

Hand it a few seconds of reference audio and it swaps the timbre — no text annotation, no per-speaker model. What eats VRAM here is never the model; it's how long a clip you feed it.

FreeVC · self-hosted

FreeVC comes from "FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion" (arXiv:2210.15418, Jingyi Li / Weiping Tu / Li Xiao), MIT-licensed at github.com/OlaWod/FreeVC. It borrows the end-to-end VITS framework for waveform reconstruction, pulls content from WavLM-Large and squeezes speaker identity out with an information bottleneck, then adds spectrogram-resize augmentation — a rescale along the mel-frequency axis, effectively formant perturbation — to scrub the content representation further. The paper reports MOS of 3.99 / 4.06 / 4.06 for seen-to-seen, unseen-to-seen and unseen-to-unseen, with speaker-similarity SMOS of 3.80 / 3.77 / 2.83, beating VQMIVC (MOS 2.31) and YourTTS (3.46) from the same period.

As a deployment target, FreeVC is absurdly small. Crack open the official freevc.pth and count: the generator is exactly 39,339,712 parameters, 150MB in fp32 — decoder 14.5M, posterior encoder 8.8M, flow 8.7M, prior encoder 7.3M. The checkpoint weighs 473MB only because it also carries both Adam momentum buffers. The real bulk is the content encoder, WavLM-Large: a 1.26GB weight file, roughly 315M parameters. Add it all up — WavLM 1.26GB plus the 150MB generator plus a 17MB GE2E speaker encoder — and you are at about 1.4GB. Load that onto a 24GB card and the remaining 22GB exists purely to absorb clip length.

One thing to be straight about: FreeVC's code has not moved since February 2023, and the only 2025 activity was two README touches. There is no v2, no rename. Officially there are exactly three checkpoints: freevc.pth, freevc-s.pth, freevc-24.pth. The install path that is still maintained today is coqui-tts (PyPI 0.27.5, Python 3.10–3.14), where it lives as voice_conversion_models/multilingual/vctk/freevc24. If you want real-time conversion, singing voice conversion or stronger non-English zero-shot, look at Seed-VC (arXiv:2411.09943), OpenVoice V2 or kNN-VC instead. But if what you want is an MIT-licensed, structurally transparent voice-conversion baseline you can train from scratch on a single card and then take apart, FreeVC is still the least painful option on the table.

01 —

Three official checkpoints, plus one maintained distribution

FreeVC has no version line — only variants. Every difference is in this table.

Version	Parameters	VRAM	Context	Notes
FreeVC (freevc.pth, 473MB)	39.3M generator + ~315M WavLM-Large	~1.4GB fp32 resident; under 3GB peak on 30s chunks	16kHz in / 16kHz out	The default. use_spk=true, so target timbre comes from the 17MB GE2E speaker encoder as a 256-dim embedding (gin_channels=256) — swap the reference clip, swap the voice, no retraining.
FreeVC-s (freevc-s.pth, 490MB)	Same generator size; inference also runs the posterior encoder enc_q	Same order as the default, ~1.4GB	16kHz in / 16kHz out	use_spk=false — timbre is read straight from the target mel, no external speaker encoder. Watch out: convert.py trims the reference with librosa.effects.trim(top_db=20) while training's data_utils.py does not, a train/inference mismatch reported in issue #96.
FreeVC-24 (freevc-24.pth)	Same architecture; decoder upsample_rates changed to [10,6,4,2] for 480x total	Identical to the 16kHz build, ~1.4GB	16kHz content features / 24kHz out	Multiplies the 50-frames-per-second WavLM rate by 480 to emit 24kHz waveform directly, segment_size 8640. Sounds more open, but content is still extracted at 16kHz — do not mistake it for true wideband capture.
freevc-nosr (configs/freevc-nosr.json)	A training recipe, not a checkpoint	Same as the default recipe; batch_size 64 fits in 24GB	16kHz in / 16kHz out	Training config with spectrogram-resize augmentation turned off. Saves close to 900GB of preprocessing intermediates, at the cost of more source-speaker timbre leaking through the content features.
coqui-tts freevc24 (voice_conversion_models/multilingual/vctk/freevc24)	Packaged model.pth is 1.33GiB — WavLM is bundled inside	~1.4GB of weights resident once loaded	16kHz in / 24kHz out	One 854MB zip, one command to convert. The important part: it mirrors the original WavLM-Large.pt rather than HuggingFace's microsoft/wavlm-large, whose weights differ slightly — the author called this out on the demo Space as something that degrades quality.

02 —

Which card to rent

FreeVC ships with fp16_run set to false — training and inference are fp32 end to end, so pick on fp32 throughput and headroom, not Tensor Cores

Auditioning voices, batch conversion with clips under 30s
RTX 3090 24GB$0.193/GPU-hour
1.4GB of weights leaves 22GB purely for attention, and this is the cheapest 24GB card on the network — about 35% under the 16GB Tesla T4 at $0.298.
Training from scratch on the paper recipe: batch_size 64, 900k steps
RTX 3090 24GB$0.193/GPU-hour
The paper trained on a single 3090. You can run configs/freevc.json unmodified — segment_size 8960, max_speclen 128, all of it.
Cutting the wall-clock time on those 900k steps
RTX 4090 24GB$0.540/GPU-hour
No AMP anywhere in FreeVC, so training is pure fp32 and the 4090's single-precision throughput is substantially higher than the 3090's, with the same 24GB — the recipe ports over untouched.
Sharding the SR augmentation preprocessing (the README splits 68–92 into six ranges)
RTX 3090 24GB x 4 on one node$0.193/GPU-hour x 4 = $0.772/hour
preprocess_sr.py is already written to shard by CUDA_VISIBLE_DEVICES, and a node takes up to 14 GPUs — run all six ranges at once and preprocessing drops from a full day to a few hours.

03 —

From boot to your first converted clip

Confirm the timbre lands with one command first, then decide whether you need the source repo

01
Boot a PyTorch instance and take the coqui-tts shortcut
If you would rather not touch the 2023 dependency set, this is the fastest way to hear the result: coqui-tts packages FreeVC-24 together with the original WavLM-Large.pt. First run pulls an 854MB zip that expands to a 1.33GiB model.pth. Trim leading and trailing silence off the reference clip — the author has stated plainly that excess silence in the reference badly hurts speaker similarity.
```
pip install coqui-tts && tts --model_name "voice_conversion_models/multilingual/vctk/freevc24" --source_wav src.wav --target_wav tgt.wav --out_path out.wav
```
02
Clone the source repo and put all three weight files where it expects them
FreeVC hardcodes its paths: WavLM-Large.pt under wavlm/, freevc.pth under checkpoints/, and the 17MB GE2E speaker encoder at speaker_encoder/ckpt/pretrained_bak_5805000.pt. Do not install requirements.txt verbatim — it pins torch==1.10.0 alongside torchvision==0.9.0, which is an incompatible pair to begin with, and nails numpy to 1.21.6. Use the image's modern torch and add the audio libraries, substituting webrtcvad-wheels for webrtcvad so you get prebuilt binaries.
```
git clone https://github.com/OlaWod/FreeVC.git && cd FreeVC && mkdir -p wavlm checkpoints && wget -O wavlm/WavLM-Large.pt https://github.com/coqui-ai/TTS/releases/download/v0.13.0_models/WavLM-Large.pt && pip install librosa scipy tqdm webrtcvad-wheels
```
03
Patch torch.load
PyTorch 2.6 flipped the weights_only default on torch.load to True — an explicitly backward-compatibility-breaking change. Both WavLM-Large.pt (which carries a cfg object) and freevc.pth (which carries optimizer state) contain non-tensor objects, so an unpatched run raises UnpicklingError. Neither utils.get_cmodel() nor utils.load_checkpoint() passes the flag, so the cheapest fix is wrapping torch.load at the top of the entry script.
```
sed -i '1i import torch, functools; torch.load = functools.partial(torch.load, weights_only=False)' convert.py
```
04
Write convert.txt and run the batch
convert.py reads a manifest of title|source_wav|target_wav lines, not command-line audio arguments — this trips up more people than anything else in the repo. Also remember WavLM runs full self-attention over the whole waveform, so VRAM grows with the square of duration: chunk long recordings to 30 seconds or less before queueing them. Stop the instance when the batch finishes and compute billing stops with it.
```
echo "demo01|wavs/src.wav|wavs/tgt.wav" > convert.txt && CUDA_VISIBLE_DEVICES=0 python convert.py --hpfile configs/freevc.json --ptfile checkpoints/freevc.pth --txtpath convert.txt --outdir outputs/freevc
```

The arithmetic, in full

Inference first. The whole stack is 1.4GB, and an RTX 3090 24GB is $0.193 per GPU-hour: boot, install, download weights, convert a batch — an hour covers it, so $0.193. Billing is per second, so if you actually used 41 minutes you pay for 41 minutes: $0.132. Egress is a rounding error — 16kHz mono 16-bit is 32KB per second, so a thousand ten-second outputs is about 320MB, and at $0.0081/GB that is $0.0026. Now reproducing the paper from scratch. Issue #75 in the repo gives a measured anchor: a Tesla V100-SXM2-16GB running VCTK did roughly 70,000 iterations in 12 hours, about 5,800 steps per hour. The recipe calls for 900k steps, and 900,000 / 5,800 is about 155 hours. On Tesla V100 32GB at $0.188/GPU-hour that is 155 x 0.188 = $29.14; on the paper's own RTX 3090 at $0.193 it is 155 x 0.193 = $29.92, and the 3090's faster fp32 means the real run is shorter. Under thirty dollars for a paper-grade voice conversion model is the most underrated thing about FreeVC. Storage is what will actually bite you, not compute. SR augmentation writes 25 copies per utterance (i from 68 to 92): the resampled wav is 32KB per second, and the WavLM feature is 1024 dims x 50 frames/sec x 4 bytes = 200KB per second, so about 232KB/sec x 25 = roughly 5.8MB per second of source audio. VCTK's ~44 hours (158,400 seconds, less after silence trimming) lands near 900GB of intermediates. At $0.414/GB-month, parking that untouched for a month is several hundred dollars. So: delete dataset/sr the moment training ends, or only run one or two of the 68–92 ranges, or switch to configs/freevc-nosr.json outright. Compute billing stops when the instance stops; storage keeps billing until it is destroyed — and on FreeVC those two numbers are two orders of magnitude apart.

04 —

FAQ

How much VRAM does FreeVC need to self-host? Is a 6GB card enough?

Resident inference weights are about 1.4GB — WavLM-Large at 1.26GB, the 39.3M-parameter generator at 150MB, the speaker encoder at 17MB — so yes, a 6GB card runs it. The ceiling is not the weights, it is clip length: WavLM does full self-attention over the whole waveform at 50 frames per second, and the attention matrix is roughly 16 heads x T squared x 4 bytes, with T = 50 x seconds. That is 144MB at 30 seconds, 576MB at a minute, 2.3GB at two minutes, and over 14GB at five — before you count the softmax intermediates. Either chunk properly, or just take a 24GB card. NexGPU's RTX 3090 24GB is $0.193/GPU-hour, cheaper than plenty of 16GB cards and billed per second, so chunking stops being a decision you have to agonise over.

How is FreeVC different from RVC or so-vits-svc, and which should I pick?

Fundamentally different shapes. RVC and so-vits-svc train a dedicated model per target voice — solid results, but every new voice means another training run. FreeVC is one-shot: train once, then hand it a reference clip and the voice changes, even for speakers never seen in training (the paper's unseen-to-unseen figures are MOS 4.06 and SMOS 2.83, so similarity does drop a notch). If you need many voices on demand, FreeVC is far less work; if you need one voice pushed to its limit, the per-speaker route wins. On NexGPU you can install both on the same box and compare directly — RTX 3090 from $0.193/GPU-hour, per-second billing, stop the instance and the charges stop.

Does FreeVC work for Chinese or other non-English languages?

The official weights were trained on VCTK's 107 English read-speech speakers at 16kHz, and WavLM-Large's own pretraining corpus is predominantly English, so non-English input produces intelligible speech but noticeably weaker prosody and speaker similarity. The right fix is fine-tuning on in-language data with configs/freevc-nosr.json, or freevc.json if you want SR augmentation — the generator is only 39.3M parameters, so this costs far less than instinct suggests. On NexGPU an RTX 3090 24GB is $0.193/GPU-hour with PyTorch images preinstalled, which puts a forty-hour fine-tune at $7.72.

Why does the converted audio still carry the source speaker's pitch?

Because FreeVC does not model F0 at all. Content comes straight from WavLM features through an information bottleneck, with no separate pitch representation to reset, so the source pitch contour leaks through with the content. Issue #27 documents exactly this on a 690k-step model: a high-pitched female source converted to a male target still tracks the source pitch. It is an architectural trade-off, not undertrained weights. For reliable cross-gender or wide-range conversion you want a method that models F0 explicitly. To A/B several approaches on the same material, spin up a 3090 on NexGPU at $0.193/GPU-hour — you will have an answer inside an hour.

FreeVC has not been updated since 2023 — is it still worth using?

The code is genuinely frozen: the last functional commit was February 2023 and 2025 brought only README edits. Frozen is not broken, though. MIT licence, 39.3M parameters, inference in a single convert.py, and a plain VITS variant underneath means there is no black box anywhere. It also has one actively maintained distribution path — voice_conversion_models/multilingual/vctk/freevc24 in coqui-tts 0.27.5, which supports Python up to 3.14 and mirrors the original WavLM weights. For real-time, singing, or stronger zero-shot, look at Seed-VC, OpenVoice V2 or kNN-VC. Whichever you land on, NexGPU's 2,000+ prebuilt images already carry PyTorch, vLLM and Whisper ASR environments, so you are comparing within minutes of boot.

How long does training FreeVC from scratch take, and what does the GPU time cost?

The paper's recipe is batch_size 64, 900k steps, a single RTX 3090, fp32 throughout. Extrapolating from the measured figure in issue #75 — a V100 16GB doing about 70,000 steps in 12 hours, roughly 5,800 steps per hour — 900k steps is about 155 hours. On NexGPU that is Tesla V100 32GB at $0.188/GPU-hour x 155 = about $29.1, or the paper's own RTX 3090 at $0.193 x 155 = about $29.9. No minimum rental, no setup fee, no quota request, billed per second, and compute billing stops the moment you stop the instance. The number to actually watch is the ~900GB of SR augmentation intermediates — destroy that volume when training ends, because storage bills at $0.414/GB-month until you do.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.