Voice Conversion / Voice Cloning

Self-host Seed-VC: clone a voice from 1 to 30 seconds of reference audio, no training

v1 ships three models at 25M, 98M and 200M params; v2 is a 67M CFM plus 90M AR pair. What actually eats VRAM is never the DiT backbone — it's the stack of upstream encoders bolted to it.

Seed-VC · self-hosted

Seed-VC comes from Songting Liu's paper Zero-shot Voice Conversion with Diffusion Transformers (arXiv:2411.09943). It uses a diffusion transformer for zero-shot voice conversion, adds an external timbre shifter during training to suppress timbre leakage, and feeds the entire reference utterance in as context for in-context learning at inference time. Hand it 1 to 30 seconds of reference audio and it clones the timbre with no training at all. The official EVAL runs 100 utterances from LibriTTS-test-clean: Seed-VC scores SECS 0.8676, WER 11.99, CER 2.92, against OpenVoice at 0.7547 / 15.46 / 4.73 and CosyVoice at 0.8440 / 18.98 / 7.29 on the same task.

The project lives at Plachtaa/seed-vc on GitHub under GPL-3.0, with close to 4k stars. Code stopped moving on 20 April 2025 and the repository was archived read-only on 21 November 2025; the author moved on to StreamVoiceAnon (ICASSP'26, Apache-2.0, real-time streaming voice anonymization and conversion). Archived does not mean unusable — the weights are still hosted on Hugging Face under Plachta/Seed-VC, and requirements.txt pins torch==2.4.0, transformers==4.46.3 and numpy==1.26.4, which makes the environment more reproducible than plenty of actively maintained projects. The CHANGELOG entry dating V2 to 2024-04-16 is a typo; it actually shipped in April 2025.

One thing to get straight before you provision anything: the 25M, 98M and 200M figures in the model table are DiT backbone parameters, not memory footprint. A running instance also keeps the content encoder resident (facebook/wav2vec2-xls-r-300m at layer 12 for the tiny tier, openai/whisper-small for the offline and singing tiers, facebook/hubert-large-ll60k at layer 18 for v2), plus the vocoder (HIFT or nvidia/bigvgan_v2_22khz_80band_256x), the CAMPPlus speaker encoder, and OpenVoice's se_db.pt (102MB) used by the timbre shifter. Add it up and each tier lands between 1.5GB and 2.5GB of weights. That is why a 25M model does not mean 25MB of VRAM.

01 —

Four official models, one job each

Parameter counts, hidden dims and layer counts come from the official model table; checkpoint sizes come from the Hugging Face Plachta/Seed-VC file listing.

Version	Parameters	VRAM	Context	Notes
seed-uvit-tat-xlsr-tiny (v1.0)	25M (hidden 384 / 9 layers)	DiT_uvit_tat_xlsr_ema.pth 142MB + XLSR-large encoder ~1.2GB + hift.pt 82MB ≈ 1.5GB of weights; 6GB is comfortable in FP16	22050 Hz, 1-30s reference	The only tier tuned for real-time conversion. Content encoder is wav2vec2-xls-r-300m, vocoder swaps to the lightweight HIFT, and real-time-gui.py pulls this one by default.
seed-uvit-whisper-small-wavenet (v1.0)	98M (hidden 512 / 13 layers)	DiT_seed_v2_uvit_whisper_small_wavenet_bigvgan_pruned.pth 440MB + Whisper-small encoder + BigVGAN ≈ 1.5GB of weights; 8GB or more recommended	22050 Hz, 1-30s reference	The default offline tier — better quality than tiny, slightly slower. inference.py auto-downloads this whenever --f0-condition is False.
seed-uvit-whisper-base (v1.0, f0-44k)	200M (hidden 768 / 17 layers)	DiT_..._whisper_base_f0_44k_bigvgan_pruned_ft_ema.pth 821MB + Whisper-small + 44.1kHz BigVGAN ≈ 2.5GB of weights; 16GB or more for chunked long-form audio	44100 Hz, 1-30s reference	Singing voice conversion with f0 conditioning. Requires --f0-condition True, official guidance is 30-50 diffusion steps, and male-to-female work usually pairs with --semi-tone-shift ±12.
hubert-bsqvae-small (v2.0)	67M (CFM) + 90M (AR)	cfm_small.pth 353MB + ar_base.pth 359MB + hubert-large-ll60k ~1.2GB + BigVGAN ≈ 2.3GB of weights; 12GB or more once --compile is on	22050 Hz, 1-30s reference	Content side swaps in ASTRAL-Quantization's binary spherical quantizer (narrow codebook 32, wide codebook 2048). The README calls it best in suppressing source speaker traits. Adds accent and emotion conversion, plus --anonymization-only to map any voice onto an average one.
StreamVoiceAnon (the author's successor)	Parameter count not published	Weights download from Hugging Face Plachta/StreamVoiceAnon; the project publishes an RTF < 1.0 target but no VRAM figure	Real-time streaming, delay configurable in frames	What Plachtaa moved to after archiving seed-vc: an ICASSP'26 paper on real-time voice anonymization and conversion, relicensed to Apache-2.0. On Windows it still needs triton-windows to stay under RTF 1.0.

02 —

Which card: pick by workload, not by capacity

Seed-VC is not a VRAM hog, so the deciding factors are latency, concurrency and sample rate rather than whether the weights fit.

Zero-shot evaluation and batch offline conversion (whisper-small-wavenet)
RTX 3090 24GB$0.193/GPU-hr
The full stack is around 1.5GB of weights, so 24GB goes entirely to long-form chunking and parallel workers — and this is the cheapest 24GB card on the network.
Real-time voice changing for streaming, meetings and game chat (xlsr-tiny)
RTX 4090 24GB$0.540/GPU-hr
The official benchmark already hits 150ms per chunk and 430ms end-to-end on an RTX 3060 Laptop; a 4090 leaves enough headroom to raise diffusion steps from 10 to 25, or run several streams without exceeding the 0.18s block time.
Single- or multi-speaker fine-tuning (train.py, batch-size 2)
Tesla T4 16GB$0.298/GPU-hr
This is literally the card the README benchmarks on — minimum 100 steps, 2 min on T4 — so you can cost the job from published numbers instead of guessing.
44.1kHz singing conversion at volume, or v2 with --compile on the AR model
RTX A6000 48GB$0.817/GPU-hr
Decoding full-length tracks through the 44k BigVGAN is the most activation-hungry path here; 48GB runs four or five singing conversions concurrently and keeps the v2 AR compile cache on the same card.

03 —

Four steps to a working Seed-VC

Python 3.10 is the recommended runtime. There is a trap in requirements.txt you need to clear in step one.

01
Provision, fix requirements.txt, then install
The first three lines of requirements.txt point at the PyTorch cu126 nightly index, while lines 5-7 hard-pin torch/torchvision/torchaudio to 2.4.0/0.19.0/2.4.0. Running pip install as-is makes pip thrash between nightly wheels and pinned versions and can leave you on a CUDA build that does not match the driver. Delete those three lines first. NexGPU's prebuilt PyTorch images already ship the driver and CUDA runtime, so SSH in and go.
```
git clone https://github.com/Plachtaa/seed-vc && cd seed-vc && sed -i '1,3d' requirements.txt && pip install -r requirements.txt
```
02
Run one zero-shot conversion; checkpoints download themselves
Checkpoints are fetched from Hugging Face on first inference. Leave --checkpoint empty and you get seed-uvit-whisper-small-wavenet. Behind a slow route to huggingface.co, prefix commands with HF_ENDPOINT to use a mirror. --fp16 defaults to True; --diffusion-steps 25 is the quality/speed balance point, drop to 4-10 for speed or push to 30-50 for quality.
```
HF_ENDPOINT=https://hf-mirror.com python inference.py --source examples/source/source_s1.wav --target examples/reference/s1p1.wav --output ./out --diffusion-steps 25 --length-adjust 1.0 --inference-cfg-rate 0.7 --fp16 True
```
03
Switch modes: singing conversion, real-time, or v2 accent transfer
Singing needs f0 conditioning and the 44k model. Real-time runs through real-time-gui.py; the official RTX 3060 Laptop settings are 10 diffusion steps, 0.18s block time, 0.04s crossfade, 2.5s extra left context and 0.02s right, where algorithm delay is roughly block time × 2 plus right context. For v2, adding --compile gives about a 6x speed-up on the AR model — straightforward on Linux with triton installed, and triton-windows==3.2.0.post13 on Windows.
```
python inference.py --source song.wav --target singer_ref.wav --output ./out --diffusion-steps 40 --f0-condition True --semi-tone-shift 0   # or: python app_vc_v2.py --compile
```
04
Fine-tune on your own data to push similarity up a notch
Clips must be 1-30 seconds or they are silently skipped; wav/flac/mp3/m4a/opus/ogg all work, speaker labels are not required but every speaker needs at least one utterance. Pick a preset from configs/presets/ matching your inference target: tiny for real-time, whisper-small-wavenet for offline, whisper-base-f0-44k for singing. Output lands at runs/<run-name>/ft_model.pth and you still supply a reference clip at inference. For v2, use accelerate launch train_v2.py, which supports multi-GPU.
```
python train.py --config ./configs/presets/config_dit_mel_seed_uvit_xlsr_tiny.yml --dataset-dir ./my_data --run-name my_speaker --batch-size 2 --max-steps 1000 --save-every 500 --num-workers 0
```

What a full evaluate-tune-deploy cycle actually costs

Line by line at NexGPU list rates. Phase one, zero-shot evaluation on a Tesla T4 16GB at $0.298/GPU-hr: pull the weights (440MB offline checkpoint + 82MB hift.pt + 102MB se_db.pt + Whisper and BigVGAN, roughly 3GB on disk) and convert 200 ten-second clips at 25 diffusion steps — download and inference together, about 40 minutes: 0.667 × $0.298 = $0.20. Phase two, fine-tune one speaker: the README's own figure is minimum 100 steps, 2 min on T4, so a full 1,000 steps is about 20 minutes — 0.333 × $0.298 = $0.10. That means going from nothing to a working ft_model.pth costs $0.30 in total. Phase three, production: real-time voice changing on an RTX 4090 24GB at $0.540/GPU-hr runs 8 × $0.540 = $4.32 for an eight-hour stream; batch 44.1kHz singing conversion on an RTX A6000 48GB at $0.817/GPU-hr with four concurrent jobs for three hours is 3 × $0.817 = $2.45. Storage bills separately: 3GB of weights at the $0.414/GB-month median works out to 3 × $0.414 = $1.24/month — and note that compute billing stops the moment you stop the instance, while storage keeps accruing until you destroy it. Egress is a rounding error: 1,000 22kHz wav files is about 0.44GB, so 0.44 × $0.0081 ≈ $0.004. Billing is metered per second and priced per hour, with no minimum, no setup fee and no quota request.

04 —

FAQ

How much VRAM does Seed-VC actually need? Is an 8GB card enough?

Yes, with room left over. Do not let the 25M/98M/200M numbers in the model table mislead you — those are DiT backbone parameters only. Real footprint has to include the content encoder (XLSR-large, Whisper-small or HuBERT-large), the vocoder (HIFT or BigVGAN) and the CAMPPlus speaker encoder, which puts the full stack between 1.5GB and 2.5GB of weights; 6-8GB runs a single stream in FP16 without complaint. What actually demands more is chunked long-form 44.1kHz singing conversion and running several streams on one card. So start evaluation on a NexGPU RTX 3090 24GB at $0.193/GPU-hr and only move up once you know what you need.

The GitHub repo is archived — is Seed-VC still worth deploying?

Yes, as long as you know what you are inheriting. The repository went read-only on 21 November 2025 and the code stopped at 20 April 2025, so there will be no new features or bug fixes. The upside is that every dependency is hard-pinned (torch 2.4.0, transformers 4.46.3, numpy 1.26.4, gradio 5.23.0) and the weights remain on Hugging Face, which makes it more reproducible than many live projects. If you need something actively maintained for real-time work, look at the author's successor project StreamVoiceAnon under Apache-2.0. You can spin up both on NexGPU in parallel, metered per second, and picking wrong costs you cents.

How does Seed-VC compare to RVC and So-VITS-SVC — can zero-shot really beat a per-speaker trained model?

The official EVAL answers this directly. On M4Singer against per-character RVCv2-f0-48k models, Seed-VC posts SECS 0.7405 vs 0.7264 and CER 19.70 vs 28.46 — it wins on both speaker similarity and intelligibility — while losing slightly on audio quality, DNSMOS OVRL 3.06 vs 3.12, which the author acknowledges outright. On the voice conversion side against three So-VITS-4.0 character models, Seed-VC leads on both SECS and WER. The trade is real: zero-shot removes per-voice training entirely at a small cost in audio quality. To A/B them yourself, put RVC and Seed-VC on one NexGPU RTX 4090 24GB at $0.540/GPU-hr.

Where does the 430ms real-time latency figure come from, and can I reproduce it?

The official test is an NVIDIA RTX 3060 Laptop GPU running seed-uvit-xlsr-tiny with 10 diffusion steps, CFG rate 0.7, max prompt length 3.0s, block time 0.18s, crossfade 0.04s, 2.5s extra left context and 0.02s right — measuring 150ms inference per chunk and 430ms end-to-end. Algorithm delay is roughly block time × 2 plus right context, with about 100ms more on the device side. The stream holds as long as inference time per chunk stays under block time, and setting CFG rate to 0.0 buys about another 1.5x. On a NexGPU RTX 4090 24GB you get far more headroom than a 3060 Laptop — spend it on lower latency or on more diffusion steps for better audio.

How much data does fine-tuning Seed-VC need, and how long does it take?

The data bar is almost comically low: one utterance per speaker is the stated minimum, clips run 1-30 seconds, wav/flac/mp3/m4a/opus/ogg all work, and speaker labels are optional — but the audio must be clean, since background music and noise visibly hurt results. On speed the README says minimum 100 steps, 2 min on T4 at the default batch-size 2; 1,000 steps is usually plenty for one speaker. The author also notes fine-tuning substantially improves speaker similarity but may slightly raise WER. NexGPU's Tesla T4 16GB is the exact card in that benchmark, at $0.298/GPU-hr, which puts the whole fine-tuning run under fifteen cents.

Seed-VC is GPL-3.0 — what should a commercial project watch out for?

GPL-3.0 is copyleft: merge Seed-VC's code into your own program and distribute it, and the derivative generally has to ship under GPL-3.0 too. The usual mitigation is to run it as a separate process or service and consume only the output audio, but where that boundary sits is a question for your counsel, not for internet folklore. Upstream dependencies carry their own terms as well — NVIDIA's BigVGAN, OpenVoice's tone-colour converter checkpoints and the CAMPPlus speaker encoder each need checking. There is also a non-legal layer: voice cloning requires the consent of the person whose voice it is. Renting a GPU on NexGPU changes none of your licensing obligations, but per-second billing does let you prove out the technical result for pocket change while legal works the rest.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.