Skip to main content
NexGPU

Speech Synthesis Model

Put CosyVoice on a card you actually control

Three seconds of reference audio clones a voice. Nine languages, 18+ Chinese dialects, 150ms bidirectional streaming. It fits on a single 3090 — VRAM was never the hard part. The dependencies are.

Sign up freeView pricing

CosyVoice is Alibaba's Tongyi speech team's LLM-based TTS stack, and the repo now lives at GitHub under QwenAudio/CosyVoice — the FunAudioLLM org was renamed to QwenAudio. Old links redirect, but update the git clone URL baked into your Dockerfile anyway. The weights did not move: HuggingFace is still FunAudioLLM, and the 1.0/2.0 checkpoints on ModelScope are still under iic. The whole project is Apache-2.0 with inference, training and deployment in the open, at roughly 22.9k stars.

The current line is Fun-CosyVoice3-0.5B-2512, released December 2025. Its llm.pt is 2.02GB, flow.pt 1.33GB, hift.pt 83.2MB — 3.44GB of fp32 weights total. You also need room for the 969MB speech_tokenizer_v3.onnx and the 28.3MB campplus.onnx. That is under 5GB for the whole system, which is far smaller than what it does. The paper scales the model to 1.5B and the training data to one million hours, but only the 0.5B weights were ever released. The 1.5B checkpoint is not public — stop hunting for a download link.

What actually stops people is the dependency chain, not the memory. Matcha-TTS is a git submodule, so forgetting --recursive earns you ModuleNotFoundError: No module named 'matcha'. ttsfrd only ships a cp310 linux_x86_64 wheel. vLLM must be pinned in lockstep with transformers. And on CosyVoice3 the code itself warns you off TensorRT fp16. On a clean GPU box that is fifteen minutes of work; on the laptop you have been installing things onto for three years it can eat a day. NexGPU has 2,000+ prebuilt images with PyTorch and vLLM ready, billed per second — get it running first, optimise second.

01 —

Which checkpoint you should actually download

Three lines coexist, with different file sizes and different sample rates — do not mix them

VersionParametersVRAMContextNotes
Fun-CosyVoice3-0.5B-25120.5B (Qwen2.5-0.5B backbone, hidden size 896)3.44GB fp32 weights (llm 2.02 + flow 1.33 + hift 0.08) | 12GB+ recommended24kHz output | 9 languages + 18 Chinese dialectsThe current line. Zero-shot cross-lingual cloning, pronunciation repair via Pinyin and CMU phonemes, instruction control over emotion, speed and volume. Note that zero-shot calls need the prompt text prefixed with You are a helpful assistant.<|endofprompt|>.
Fun-CosyVoice3-0.5B-2512 RL (llm.rl.pt)0.5B, same architecture, different LLM checkpointOne extra 2.02GB weight file | runtime footprint identical to base24kHz output | same base modelThe RL post-trained variant, shipped inside the same directory as a separate llm.rl.pt. 0.81% CER on test-zh against 1.21% for the base, and 1.68% WER on test-en. If you care about pronunciation accuracy, load this one.
CosyVoice2-0.5B0.5B2.55GB fp32 weights (llm 2.02 + flow 0.45 + hift 0.08) | 8GB+ recommended24kHz output | 25Hz token rateThe December 2024 25Hz release, with the most mature ecosystem and stable vLLM support since May 2025. 1.45% CER in Chinese, 2.57% WER in English, 75.7% speaker similarity. Pick it when you want existing tutorials to match your code.
CosyVoice-300M / -SFT / -Instruct300MFar smaller than the 0.5B line | an 8GB card is plenty22.05kHz output | 50Hz token rateThe 1.0 generation. Watch the sample rate: 22050, not 24000, so its output will not cut cleanly against anything from 2.0 onward. SFT ships preset speakers via list_available_spks(); Instruct takes natural-language direction. Not where a new project should start.
CosyVoice 3 (1.5B, paper only)1.5B— (weights not released)Paper: arXiv 2505.17589The paper scales from 0.5B to 1.5B and the corpus to one million hours, but only 0.5B was open-sourced. The realistic route to 1.5B-class quality is post-training the 0.5B on your own data, not waiting for a checkpoint drop.

02 —

Match the card to the job — a 0.5B model does not need H100 money

CosyVoice is bottlenecked on latency and concurrency, not on capacity

  • Get it running, audition zero-shot cloning, evaluate quality

    RTX 3090 24GB$0.193/GPU-hour

    The cheapest 24GB on the network — less than the 16GB Tesla T4 at $0.298 and far faster. It swallows 3.44GB of weights plus the ONNX tokenizer without thinking, which makes it the right place to fight the dependency chain.

  • Production streaming TTS with vLLM + TensorRT resident

    RTX 4090 24GB$0.540/GPU-hour

    Ada's fp16 and TensorRT throughput are the best available at this price, and 24GB leaves a genuinely useful KV cache pool for vLLM, holding first-chunk latency in the low hundreds of milliseconds.

  • High-concurrency batch dubbing, Triton + TRT-LLM multi-stream

    A100 PCIE 80GB$0.824/GPU-hour

    The official Triton benchmark hits RTF 0.0501 at LLM batch 16 on a single L20; 80GB lets you raise trt_concurrent and the vLLM memory pool at the same time and absorb dozens of concurrent streams on one card.

  • Fine-tuning your own voices, training flow and the HiFiGAN vocoder

    RTX A6000 48GB$0.817/GPU-hour

    There are already repo issues from people hitting the wall training HiFiGAN on 16GB. Training costs far more than inference, 48GB is a comfortable floor, and it prices within a cent of the A100 80GB if you'd rather switch.

03 —

From bare machine to first waveform

Four steps, commands taken straight from the repo

  1. 01

    Boot a GPU and clone the repo (with --recursive)

    Pick a PyTorch prebuilt image on the NexGPU console, start an RTX 3090 or 4090, and SSH in. Matcha-TTS is a submodule — skipping the recursive clone guarantees a failure later. Install sox while you are here, since torchaudio needs it for I/O.

    git clone --recursive https://github.com/QwenAudio/CosyVoice.git && cd CosyVoice && git submodule update --init --recursive && sudo apt-get install -y sox libsox-dev
  2. 02

    Create a Python 3.10 environment and install dependencies

    It has to be 3.10 — the only ttsfrd wheel published is cp310 linux_x86_64. Failing to install ttsfrd is not fatal: the project falls back to WeTextProcessing for text normalisation, and you lose some accuracy on numbers and symbols.

    conda create -n cosyvoice -y python=3.10 && conda activate cosyvoice && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
  3. 03

    Pull the weights and run your first zero-shot clone

    ModelScope or HuggingFace both work. Remember that CosyVoice3 prompt text needs the <|endofprompt|> prefix — this is the call-signature change from 2.0, and following an old 2.0 tutorial verbatim produces very strange audio. For cross-lingual synthesis, tag the target text with <|en|>, <|zh|>, <|ja|> and friends.

    python -c "from modelscope import snapshot_download; snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')" && export PYTHONPATH=third_party/Matcha-TTS && python example.py
  4. 04

    Go to production: vLLM acceleration or Triton multi-stream

    vLLM and transformers must be pinned as a matched pair, or registering CosyVoice2ForCausalLM throws. One critical difference: the official example uses fp16=True for CosyVoice2 but fp16=False for CosyVoice3 — the code explicitly warns that the DiT TensorRT fp16 engine has performance problems. Do not flip it on out of optimism. For more throughput, use the docker compose stack in runtime/triton_trtllm.

    pip install vllm==v0.11.0 transformers==4.57.1 numpy==1.26.4 && python vllm_example.py

What an hour of finished audio actually costs

Work from the official offline benchmark on a single L20: RTF 0.0562 at LLM batch 8, meaning one second of audio costs 0.0562 seconds of compute. One GPU-hour = 3600 ÷ 0.0562 ≈ 64,060 seconds ≈ 17.8 hours of finished audio. On a NexGPU RTX 4090 24GB at $0.540/GPU-hour, that is $0.540 ÷ 17.8 ≈ $0.030 per audio-hour. Without batching (batch 1, RTF 0.1091) a GPU-hour yields about 9.2 audio-hours, or $0.059 per audio-hour. Applied to real work: a 100,000-word audiobook at roughly 250 words per minute is about 6.7 hours of audio, which is roughly $0.20 of GPU time at batch 8, plus a one-off weight download. Drop to an RTX 3090 at $0.193/GPU-hour and the unit cost falls to about a third of that — note the RTF above was measured on an L20, so real throughput will shift with the card, though the order of magnitude holds. Billing is metered per second: stop the instance and compute charges stop immediately, while storage keeps accruing until you destroy it (median $0.414/GB-month, and this whole model is under 5GB). No minimum, no setup fee, no quota request.

04 —

Frequently Asked Questions

How much VRAM does CosyVoice actually need to self-host?

Less than most people assume. CosyVoice3-0.5B's three weight files total 3.44GB in fp32 (llm.pt 2.02GB, flow.pt 1.33GB, hift.pt 83.2MB); CosyVoice2-0.5B totals 2.55GB. Add the speech_tokenizer ONNX that runs on GPU (969MB for v3), activations and the CUDA context, and single-stream inference fits in 8GB — budget 12GB or more once vLLM wants a KV cache pool. Which is exactly why the RTX 3090 24GB at $0.193/GPU-hour on NexGPU is this model's best home: 24GB removes the question entirely.

What is the difference between CosyVoice3 and CosyVoice2, and which should I use?

CosyVoice3 brings a new speech tokenizer trained with supervised multi-task learning across ASR, emotion recognition, language ID and audio event detection, scales training data from ten thousand to one million hours, and adds an RL post-trained checkpoint — 0.81% CER on test-zh against the base model's 1.21%. CosyVoice2 wins on ecosystem maturity, tutorial compatibility and vLLM support that has been stable since May 2025. The APIs differ too: CosyVoice3 drops the load_jit argument and requires the <|endofprompt|> prefix for zero-shot prompts. Both are 0.5B, so spin up one 3090 on NexGPU and run your own voice samples through both for under a dollar.

Can I use CosyVoice commercially?

The repo is Apache-2.0, one of the most permissive open licences there is — commercial use, modification and closed-source redistribution are all allowed as long as you retain the licence and copyright notice. What the licence cannot cover is where your voices come from: zero-shot cloning means any audio clip can be a reference, but the voice rights of the person cloned, and the synthetic-speech disclosure rules in your jurisdiction, are on you. That is precisely why many teams self-host rather than call a third-party API — the reference audio never leaves their own machine. On NexGPU the instance is yours alone, and weights and audio stay on your own volume.

I get No module named 'matcha' and ttsfrd won't install. Now what?

Both are well-worn problems. The matcha error means the Matcha-TTS submodule was never pulled: run git submodule update --init --recursive, then export PYTHONPATH=third_party/Matcha-TTS. ttsfrd only publishes a cp310 linux_x86_64 wheel (ttsfrd-0.4.2-cp310-cp310-linux_x86_64.whl), so the wrong Python or OS simply cannot install it; it also needs git-lfs to fetch resource.zip, and without lfs you get an empty file that will not unzip. Neither blocks audio generation — the project falls back to WeTextProcessing. NexGPU's Ubuntu and PyTorch prebuilt images already satisfy these prerequisites, which cuts most of the debugging out.

I enabled TensorRT fp16 on CosyVoice3 and it got worse. Why?

That is not your mistake, it is a known issue. When load_trt and fp16 are both set, the code prints a warning that the DiT TensorRT fp16 engine has performance problems and should be used with caution. The official vllm_example.py reflects this: fp16=True for CosyVoice2, fp16=False for CosyVoice3. So the correct way to speed up CosyVoice3 is load_vllm=True plus load_trt=True while staying in fp32, or moving to runtime/triton_trtllm outright. Combination testing like this is exactly what per-second billing is for: take an RTX 4090 at $0.540/GPU-hour, benchmark each configuration, then stop the box.

The docs claim 150ms first-chunk latency. Why can't I hit it?

150ms is the model-side latency of the bidirectional streaming architecture under ideal conditions. The official Triton measurements on a single L20 tell the fuller story: at 4 concurrent streams, average first-chunk latency is 750.42ms, with P50 at 740.31ms, P90 at 941.05ms and P99 at 1002.37ms. The gap is queueing, text normalisation, reference-audio encoding and network round trips — none of which live inside that 150ms. To approach the floor, cut concurrency, warm the engines, and cache speaker embeddings for reuse. NexGPU has 1,175 verified rentable nodes across 51 countries and regions, so putting inference near your users often saves more milliseconds than any amount of tuning.

More in Speech synthesis

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.