Skip to main content
NexGPU

Voice Conversion / Cloning

Self-hosting DiffVC: an old 32GB card is plenty — the 2022 dependency pins are the hard part

Huawei Noah's Ark Lab's official ICLR 2022 Oral implementation. About 126M parameters and under 0.6GB of weights for the whole inference stack, with peak VRAM around 2GB for a 10-second clip at 30 sampling steps.

Sign up freeView pricing

DiffVC is the official implementation of "Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme" (ICLR 2022, Oral), living in the DiffVC subdirectory of huawei-noah/Speech-Backbones under the MIT licence. It does one-shot any-to-any conversion: hand it a source utterance plus a few seconds of reference audio from the target speaker and it swaps the timbre — neither speaker needs to have appeared in the training set. The architecture is two-stage: an 8.46M-parameter "average voice" encoder collapses the mel-spectrogram into a speaker-stripped intermediate, then a 117.8M-parameter 2D U-Net diffusion decoder reverse-diffuses a new mel conditioned on the target speaker embedding, and HiFi-GAN turns that mel into a waveform. DiffVC itself totals roughly 126M parameters and about 0.5GB of fp32 weights — the notebook prints the exact figure via generator.nparams.

The scarce resource here is not VRAM, it is patience. One conversion of a 10-second clip at n_timesteps=30 with mode='ml' peaks around 2GB; a 30-second utterance lands at 3-4GB. The U-Net treats the 80×T mel as a 2D image, so memory grows roughly linearly with clip length — the only realistic way to OOM is feeding it a multi-minute recording in one shot. The pain is entirely environmental. The shipped requirements.txt pins torch==1.7.1, torchaudio==0.5.1, librosa==0.6.0 and numpy==1.19.0; torchaudio 0.5.1 is not even the matching release for torch 1.7.1, and librosa 0.6 will not install against today's numba/numpy. Worse, the torch 1.7.1 cu110 wheel only compiles up to sm_80, so RTX 3090/A10/A6000 (sm_86), RTX 4090 (sm_89), RTX 5090 (sm_120) and H100/H200 (sm_90) all throw "no kernel image is available" or an architecture-mismatch warning.

So is DiffVC still worth deploying? It depends on what you need. Almost every later diffusion VC system grew out of it — AAAI 2024's DDDM-VC credits DiffVC directly for its diffusion code in the acknowledgements, but ships under CC BY-NC-SA 4.0, which rules out commercial use. The louder alternative, Seed-VC, is GPL-3.0 and its repository is now a public archive. DiffVC, meanwhile, lists all three components — the model, HiFi-GAN, and the speaker encoder — as MIT in its THIRD_PARTY_NOTICE, and Speech-Backbones has never been archived. For a baseline you can read, modify, and ship commercially, it is still the least painful starting point, provided you spend an hour modernising the pins.

01 —

The checkpoint list: DiffVC is four files, not one

A single conversion needs the VC model, the average-voice encoder, the vocoder, and the speaker encoder all in place

VersionParametersVRAMContextNotes
Diff-LibriTTS (vc_libritts_wodyn.pt)126M (8.46M encoder + 117.8M diffusion decoder)fp32 weights ~0.5GB; ~2GB peak for a 10s clip at 30 steps22.05kHz · 80 mel · n_fft 1024 · hop 256 · n_timesteps=30The default checkpoint, trained on roughly 1,100 LibriTTS speakers, and the one to start from for any-to-any work. The "wodyn" in the filename is the winning conditioning ablation from the paper: speaker embedding plus the noised target mel, without the full diffusion trajectory.
Diff-VCTK126M, identical architecture to the LibriTTS buildfp32 weights ~0.5GB, same runtime footprint22.05kHz · 80 mel · hop 256Trained on VCTK's 109 speakers, 99 of which the paper holds out as unseen. Better fitted to clean studio recordings and British accents; the conditioning ablation reports wodyn pushing speaker similarity to 46.7% on VCTK.
Average-voice encoder enc.pt (FwdDiffusion)8.46M (MelEncoder 6.84M + PostNet 1.62M)fp32 ~34MB6 Transformer layers, 192 channels, 768 filters, 2 heads, relative window 4You cannot train the decoder without it. Either drop the official LibriTTS build into logs_enc/, or run Montreal Forced Aligner to produce TextGrids, build average-voice mels with get_avg_mels.ipynb, and run train_enc.py yourself. In train_dec.py the Adam optimiser is attached only to model.decoder.parameters() — the encoder stays frozen throughout.
HiFi-GAN universal_v1 vocoder~14Mfp32 ~56MB22.05kHz output; mel filterbank fmin=0, fmax=8000DiffVC only produces mels — every waveform comes from here. Taken from jik876/hifi-gan's universal checkpoint, MIT licensed. Note the mel basis caps at 8kHz, so sibilance detail above 8kHz was never in the modelled band to begin with.
GE2E speaker encoder, checkpts/spk_encoder/pretrained.pt~1.4M (3-layer LSTM, hidden 256, 256-d embedding)fp32 ~6MB16kHz · 40 mel · 25ms window / 10ms hop · 1.6s partialsThe only weight that ships in the repo rather than on Google Drive, borrowed from Real-Time-Voice-Cloning. It runs a completely separate front end: webrtcvad silence trimming, normalisation to -30 dBFS, resampling to 16kHz. Change sample rates and you have to touch both pipelines, not one.

02 —

Picking a GPU: compute capability matters more than VRAM here

DiffVC is fp32 end to end — match the card's architecture to the torch version you intend to run

  • Auditioning quality, or batch-converting a few hundred to a few thousand clips under 10 seconds

    Tesla V100 32GB$0.188/GPU-hour

    The cheapest card on the list and still 32GB. sm_70 sits inside the architecture list of the official torch 1.7.1+cu110 wheel, making it one of the few cards where copying requirements.txt verbatim actually works.

  • Reproducing the official decoder training run (batch_size=32, train_frames=128, full fp32)

    RTX 3090 24GB$0.193/GPU-hour

    Adam only optimises the 117.8M-parameter decoder, so weights plus gradients plus first and second moments pin about 1.9GB and the rest is activations. 24GB takes the default config; just remember sm_86 needs torch 1.8+/cu111 or newer.

  • Converting 3-minute-plus recordings in one pass without chunking, or pushing batch_size to 64

    RTX A6000 48GB$0.817/GPU-hour

    U-Net memory scales with mel frame count, so 48GB of headroom saves you writing chunk-and-stitch logic — and removes the timbre drift that shows up at segment boundaries.

  • Modernising to torch 2.x and running 30-step reverse diffusion as a resident batch service

    RTX 4090 24GB$0.540/GPU-hour

    The paper reports GPU RTF around 0.5 for ML-30 and around 0.1 for ML-6; Ada only improves on that. 24GB comfortably keeps DiffVC, HiFi-GAN and the speaker encoder resident with room for concurrency.

03 —

Four steps to DiffVC on NexGPU

From spinning up an instance to hearing your first conversion, under twenty minutes of GPU time

  1. 01

    Spin up a CUDA instance and clone the repo

    Pick a card at console.nexgpu.net — a Tesla V100 32GB at $0.188/GPU-hour is the right first move — and grab a prebuilt PyTorch image from the 2,000+ on offer. Billing is metered per second with no minimum and no quota request, and SSH, Jupyter, web terminal, REST API and CLI are all available. DiffVC's official entry point is a notebook, so Jupyter is the path of least resistance. Note that the repo is all of Speech-Backbones; DiffVC is one subdirectory alongside Grad-TTS and SPIRAL.

    git clone https://github.com/huawei-noah/Speech-Backbones.git && cd Speech-Backbones/DiffVC
  2. 02

    Do not install requirements.txt as written

    Those pins are frozen at 2022: torch==1.7.1 paired with torchaudio==0.5.1 (not actually a matching pair), plus librosa==0.6.0 and numpy==1.19.0, neither of which will install today. Install current versions and patch exactly two call sites: mel_basis = librosa_mel_fn(22050, 1024, 80, 0, 8000) in inference.ipynb must become the keyword form librosa.filters.mel(sr=22050, n_fft=1024, n_mels=80, fmin=0, fmax=8000), and librosa.core.load / librosa.core.stft become librosa.load / librosa.stft. The model code itself is plain PyTorch plus einops and runs on torch 2.x unchanged.

    pip install torch torchaudio einops 'librosa>=0.10' numpy scipy webrtcvad tgt tqdm
  3. 03

    Fetch two checkpoints — the third ships with the repo

    checkpts/spk_encoder/pretrained.pt is already there. What you need are the conversion model under checkpts/vc/ (LibriTTS Drive ID 18Xbme0CTVo58p2vOHoTQm8PBGW7oEjAy, VCTK 12s9RPmwp9suleMkBCVetD8pub7wsDAy4) and the universal HiFi-GAN vocoder under checkpts/vocoder/ (Drive ID 10khlrM645pTbQ4rc2aNEYPba8RFDBkW-). Unpack the vocoder so that checkpts/vocoder/ contains both config.json and generator — the notebook reads those exact filenames. If you plan to retrain the decoder, the encoder weight is Drive ID 1JdoC5hh7k6Nz_oTcumH0nXNEib-GDbSq and belongs in logs_enc/.

    pip install gdown && mkdir -p checkpts/vc checkpts/vocoder && gdown 18Xbme0CTVo58p2vOHoTQm8PBGW7oEjAy -O checkpts/vc/vc_libritts_wodyn.pt
  4. 04

    Run a conversion, then fix the bug in the notebook

    Inference is two forward passes: DiffVC emits a mel, HiFi-GAN emits the waveform. Source and reference audio each go through get_mel, and the reference additionally goes through get_embed for its 256-dimensional speaker vector. Once it runs, go back to the conversion cell: mel_synth_np and mel_source_np are both assigned from mel_, but the second one should read mel_source. The result is that the mel_spectral_subtraction call immediately after subtracts the synthesised mel from itself, so the noise-suppression step does nothing at all. Change it to mel_source.cpu().detach().squeeze().numpy() and listen again — the hiss drops noticeably.

    mel_encoded, mel_ = generator.forward(mel_source, mel_source_lengths, mel_target, mel_target_lengths, embed_target, n_timesteps=30, mode='ml')

The arithmetic: 1,000 clips in two hours vs. a week reproducing the paper

Start with the cheap end. The paper reports GPU RTF around 0.5 for ML-30, so a 10-second clip costs roughly 5 seconds of GPU time. A thousand 10-second clips is about 5,000 seconds of pure conversion; add mel and speaker-embedding preprocessing plus model loading and call it 2 hours. On a Tesla V100 32GB that is 2 × $0.188 = $0.376. Now the heavy end. train_dec.py defaults to epochs=110, batch_size=32 and learning_rate=1e-4. Budget 72 hours (3 days) on an RTX 3090 24GB: 72 × $0.193 = $13.90. Data is the bigger line item. The paper trained on roughly 1,100 speakers, and LibriTTS train-clean-100 plus train-clean-360 comes to exactly 1,151 speakers and about 245 hours. Resampled to 22.05kHz that is roughly 39GB of wavs, plus roughly 24GB of precomputed float32 mels at 80 × T, with speaker embeds negligible — about 65GB. At $0.414/GB-month, one week is 65 × 0.414 × 7/30 ≈ $6.28. Then there is the line everyone forgets. train_dec.py sets save_every=1, so every epoch writes a full state_dict of about 0.5GB. Across 110 epochs that is 55GB, or 55 × 0.414 × 7/30 ≈ $5.31 for a week of storage. Total for the week: $13.90 + $6.28 + $5.31 ≈ $25.49, with compute accounting for barely more than half. Which is the NexGPU billing detail worth internalising: compute billing stops the moment the instance stops, but storage keeps billing until the volume is destroyed. When training finishes, keep the last few checkpoints and delete the data volume before you walk away.

04 —

FAQ

How much VRAM does DiffVC need to self-host? Is a 24GB card enough?

For inference, comfortably. DiffVC is about 126M parameters, and together with HiFi-GAN and the speaker encoder the fp32 weights total under 0.6GB. A 10-second clip at 30 ml steps peaks around 2GB; a 30-second utterance runs 3-4GB. Because the U-Net treats the 80×T mel as a 2D image, memory scales with clip length, so your real ceiling is how much audio you feed at once. Training is different: the defaults of batch_size=32, train_frames=128 (about a 1.49-second mel crop) and full fp32 fit 24GB with little to spare, and a 16GB card needs batch_size around 8-12. On NexGPU that maps to a Tesla V100 32GB at $0.188/GPU-hour for inference and an RTX 3090 24GB at $0.193/GPU-hour for training — metered per second, so measuring beats estimating.

Has DiffVC been superseded? Is there still a reason to deploy it?

It has not been deprecated — the Speech-Backbones repository still is not archived. Later work is stronger but licensed tighter: AAAI 2024's DDDM-VC explicitly credits DiffVC for its diffusion code yet ships under CC BY-NC-SA 4.0, which rules out commercial use. Seed-VC is the noisier alternative, with 25M/98M/200M v1 checkpoints and a 157M hubert-bsqvae-small in v2, but it is GPL-3.0 and its repo is now a public archive. DiffVC's THIRD_PARTY_NOTICE lists the model, HiFi-GAN and the speaker encoder as MIT, line by line — the easiest of the family to actually ship. For baselines, architecture surgery, or anything commercial, it remains the cheapest starting point. An evening on a single NexGPU card costs less than an evening reading GitHub issues.

I installed requirements.txt and nothing runs — librosa and torch keep erroring. What now?

Those pins are from 2022. torchaudio==0.5.1 was never the matching release for torch==1.7.1, and librosa==0.6.0 simply will not build against current numba/numpy. Install current torch and librosa, then patch two call sites: mel_basis = librosa_mel_fn(22050, 1024, 80, 0, 8000) becomes the keyword form librosa.filters.mel(sr=22050, n_fft=1024, n_mels=80, fmin=0, fmax=8000), and librosa.core.load / librosa.core.stft become librosa.load / librosa.stft. The model code is plain PyTorch plus einops and needs no changes under torch 2.x. NexGPU ships 2,000+ prebuilt images including PyTorch, so most of that layer is already done for you.

Why does torch 1.7.1 throw "no kernel image" on an RTX 4090, 5090 or H100?

Because the torch 1.7.1 cu110 wheel compiles only up to sm_80. Tesla V100 is sm_70, Tesla T4 is sm_75 and A100 is sm_80 — all covered. RTX 3090, A10 and A6000 are sm_86, RTX 4090 is sm_89, H100 and H200 are sm_90, and RTX 5090 is sm_120 — none covered. Either stay on V100/T4 and run the original pinned stack, or upgrade torch, which is the better option since DiffVC's model code runs unmodified on torch 2.x. One more thing: this is a 126M-parameter model with a 2GB peak footprint, so renting an H100 SXM 80GB at $3.582/GPU-hour to run it is pure waste. NexGPU carries 75 GPU models across 2,498 cards in 51 countries and regions — matching the tier matters far more than buying the top one.

Does DiffVC work for Chinese? How well does it hold up on a Mandarin corpus?

Voice conversion operates at the acoustic level and never touches text, so Chinese audio will produce output. Set expectations, though: the released weights are trained on LibriTTS and VCTK, both English, and the average-voice encoder's supervision comes from Montreal Forced Aligner phoneme alignments on English. Mandarin tone and syllable structure are not in that prior, so cross-lingual conversion carries an audible accent drift. Doing it properly means retraining on your own multi-speaker Mandarin corpus: arrange the data as wavs/mels/embeds directories split by speaker, take or train the encoder, then run train_dec.py for the decoder. An RTX A6000 48GB at $0.817/GPU-hour on NexGPU is the least painful place to do that — 48GB means you never have to cut batch size to fit.

The converted audio has background hiss. Is that the model?

Half of it is, half of it is the notebook. The diffusion decoder does leave residual noise in the mel, which is why mel_spectral_subtraction exists — but in inference.ipynb both mel_synth_np and mel_source_np are assigned from mel_, where the second should be mel_source, so the subtraction runs the synthesised mel against itself and does nothing. Fix that line first. If it is still not clean, raise n_timesteps above 30 (the paper takes the EM solver to 100 steps and compares ML and PF at 6 and 30; more steps means cleaner output and linearly worse RTF, and train_dec.py's own validation inference uses 100), or try the deterministic mode='pf' sampler. Tuning like this means renting a card and listening repeatedly, which is why per-second billing with no minimum and no quota request matters — ten parameter sweeps cost less than a coffee. And if V100 supply is tight, a Tesla P40 24GB at $0.214/GPU-hour is a ready stand-in for this fp32 pipeline.

More in Voice cloning and conversion

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.