Skip to main content
NexGPU

Speech synthesis / voice cloning

Self-hosting OpenVoice: a 131MB tone color converter that clones six languages on a budget card

OpenVoice is one of the last voice cloning stacks that is genuinely small — MIT licensed, purely feed-forward, under 350MB of weights end to end. Here is what actually separates V1 from V2, what your reference audio must look like, and the dependency traps you will hit.

Sign up freeView pricing

OpenVoice comes from Qin Zengyi (MIT), Zhao Wenliang and Yu Xumin (Tsinghua), and Sun Xin (MyShell), published as arXiv:2312.01479, "OpenVoice: Versatile Instant Voice Cloning". The code lives at github.com/myshell-ai/OpenVoice with 37k stars, and both V1 and V2 are MIT licensed — free for commercial and research use. Architecturally it has nothing in common with the LLM-style TTS models that followed it: a base speaker TTS reads the text, then a separate tone color converter repaints that audio with the reference speaker's timbre. That split explains every strength and every limitation. The converter is a feed-forward flow model with no autoregressive token loop, so latency is stable and predictable — but it only handles timbre. Accent, emotion, and rhythm all come from the base speaker.

The numbers are absurdly small. V2's converter checkpoint.pth is 131,320,490 bytes (~125MiB), and its config declares a 22050Hz sample rate, hop 256, hidden dim 192, 6 layers, gin_channels 256, and HiFi-GAN upsampling of [8,8,2,2] — classic VITS lineage. V2 swaps the base speaker to MeloTTS, one fp32 checkpoint per language: 207,770,124 bytes for Chinese, 207,602,918 bytes for English v3. So the most common path, cloning a voice into Chinese, totals roughly 340MB of weights. Issue #48 in the repo reports about 3GiB of system RAM and about 1GiB of VRAM at init. The practical conclusion: renting an H100 for OpenVoice is pure waste. The real question is never "will it fit" but "how many concurrent workers fit on one card".

Now the honest part. The OpenVoice repo's last code push was 19 April 2025, and V2's weight bundle is still checkpoints_v2_0417.zip from April 2024. There is no V3 — MyShell's HuggingFace org has only published DreamVoice and ShellAgent since. This line has settled. The main consequence is frozen dependencies: requirements.txt pins numpy==1.22.0, librosa==0.9.1, gradio==3.48.0, faster-whisper==0.9.0, with Python 3.9 as the official environment, which simply will not install against today's NumPy 2.x. If you want richer prosody and emotion, Apache-2.0 CosyVoice (23k stars, still actively pushed) is the other road. If you want cheap, controllable, latency-stable synthesis at tens of thousands of clips per batch, OpenVoice is still one of the best value-per-dollar options available.

01 —

Versions and weights: exactly which files you need

V2 is not one model — it is a MeloTTS base speaker, a tone color converter, and a set of speaker embeddings

VersionParametersVRAMContextNotes
OpenVoice V2 tone color converter (checkpoints_v2_0417)converter/checkpoint.pth = 131,320,490 B (~125MiB, fp32)~0.13GB resident weights; ~1GB in practice with CUDA context and activations22.05kHz mono output, zero_g=true, 6 layers / hidden 192 / gin 256The heart of the pipeline: it changes timbre only, never content. V1 and V2 converters are nearly identical in size but the weights differ — do not mix them.
MeloTTS base speakers (EN / ES / FR / ZH / JP / KR)~208MB fp32 each; Chinese 207,770,124 B, English v3 207,602,918 B~0.21GB per language; ~1.25GB with all six residentThe Chinese base speaker code-switches natively — the official demo text mixes Chinese and EnglishPronunciation, accent, speed and phrasing in V2 come entirely from here. OpenVoice itself contributes none of it.
V2 source speaker embeddings, base_speakers/ses (11 files)~1.7KB .pth each — a 256-dim gin vectorNegligibleen-newest / en-us / en-br / en-au / en-india / en-default / es / fr / zh / jp / krMust match the MeloTTS speaker_id you synthesised with. Mismatch it and the converted voice drifts and sounds mechanical.
OpenVoice V1 (checkpoints_1226)EN and ZH base speakers 160,467,309 B each; converter 131,327,338 B; 452MB total on HFIssue #48 measured ~1GiB VRAM plus ~3GiB RAM initialising BaseSpeakerTTSNative base speakers for English and Chinese onlyThe only release with emotional style tags (whispering / cheerful / sad and others). V2 dropped that style control, so if you need emotion you stay on V1.
Target speaker embedding, target_seOutput of se_extractor.get_se(), a 256-dim tensorNegligibleReference audio must be clean, single-speaker, long enough, and free of long silencesExtract once, save it as a .pth, reuse it. Re-running VAD segmentation on every synthesis is where your wall-clock time actually goes.

02 —

Which card to rent: pick by concurrency, not capacity

Under 350MB of weights means VRAM capacity was never the bottleneck — per-process CUDA context and single-card throughput are

  • First run: install, pull weights, extract one voice, synthesise a test clip

    Tesla V100 32GB$0.188/GPU-hr

    The cheapest tier we rent, with 32GB that is comically oversized for this stack, and Volta is stably supported by every PyTorch release — exactly what you want while wrestling that Python 3.9 dependency set.

  • Day-to-day development: Gradio tuning, swapping reference clips to A/B the result

    RTX 3090 24GB$0.193/GPU-hr

    Ampere is the least painful target for current CUDA and torch builds, for half a cent more than the V100 — the right trade when your loop is edit one line, restart the process.

  • Production API: dozens of concurrent real-time streams with low, stable time-to-first-audio

    RTX 4090 24GB$0.540/GPU-hr

    The converter is feed-forward, so per-request compute is tiny; the 4090's high clock throughput saturates concurrency, and 24GB holds a dozen-plus resident worker processes.

  • All six languages resident plus a bulk offline dubbing pipeline

    RTX A6000 48GB$0.817/GPU-hr

    Six MeloTTS base speakers plus the converter is only ~1.38GB of weights; what actually consumes VRAM is 0.3–0.5GB of CUDA context per worker, and 48GB lets you stop counting processes.

03 —

Four steps from empty instance to your first cloned clip on NexGPU

Each step flags the place people actually get stuck

  1. 01

    Boot an instance and build a Python 3.9 environment

    Launch from a NexGPU PyTorch prebuilt image and SSH in. Build a Python 3.9 environment as the project requires — requirements.txt pins numpy==1.22.0 and librosa==0.9.1, which fail to build on Python 3.11 with NumPy 2.x. This is by far the most common first failure.

    conda create -n openvoice python=3.9 -y && conda activate openvoice && git clone https://github.com/myshell-ai/OpenVoice.git && cd OpenVoice && pip install -e .
  2. 02

    Install MeloTTS and download the V2 weights

    V2's base speaker lives in a separate repo and must be installed on its own; Japanese additionally needs the unidic dictionary. Unzipping checkpoints_v2_0417.zip gives you converter/ and base_speakers/ses/. If Silero VAD fails to download, drop the zip into ~/.cache/torch/hub/snakers4_silero-vad_master by hand and move on.

    pip install git+https://github.com/myshell-ai/MeloTTS.git && python -m unidic download && wget https://myshell-public-repo-host.s3.amazonaws.com/openvoice/checkpoints_v2_0417.zip && unzip checkpoints_v2_0417.zip
  3. 03

    Extract the target timbre from your reference clip

    Load the converter, then call se_extractor.get_se(); vad=True runs Silero to segment out silence. The reference must be clean, single-speaker, long enough, and without long blank stretches. Note that intermediates are cached under processed/ — if you change the audio but keep the filename, delete that folder or you will keep hearing the previous take.

    target_se, audio_name = se_extractor.get_se('resources/example_reference.mp3', tone_color_converter, vad=True)
  4. 04

    Synthesise with MeloTTS, then convert the timbre

    MeloTTS reads the text into tmp.wav, then convert() takes the source embedding (the matching file from base_speakers/ses) and your target embedding. Watch the message argument: the official demo passes "@MyShell", which embeds a wavmark watermark into the output. Decide what that parameter should be before you put a service in front of it.

    tone_color_converter.convert(audio_src_path=src_path, src_se=source_se, tgt_se=target_se, output_path=save_path, message="@MyShell")

The real bill for ten thousand dubbed clips

Get it working on a Tesla V100 32GB at $0.188/GPU-hr: about 20 minutes to build the environment and install MeloTTS, under a minute to pull the 131MB converter and the 208MB Chinese base speaker, then half an hour extracting a voice and testing. Call it one hour: 1 × $0.188 = $0.188. For the batch, switch to an RTX 4090 at $0.540/GPU-hr. Take ten thousand clips averaging 12 seconds — 120,000 seconds, about 33.3 hours of audio. At a measured 8x real time (the feed-forward converter is usually faster; benchmark your own dependency set), that is 33.3 ÷ 8 ≈ 4.17 GPU-hours: 4.17 × $0.540 = $2.25. Output is 22.05kHz 16-bit mono, 44,100 bytes per second, so 120,000 seconds is about 5.3GB of wav; with 0.6GB of weights that is 5.9GB at the $0.414/GB-month median, or $2.44 a month, and pulling the 5.3GB back down costs 5.3 × $0.0081 = $0.043 at the median egress rate. Compute plus egress: $0.188 + $2.25 + $0.04 ≈ $2.48. Destroy the volume the same day and the storage line rounds to nothing. Billing is metered per second and priced per hour, compute billing stops the moment the instance stops — only storage keeps accruing until you destroy it. No minimum, no setup fee, no quota request.

04 —

Frequently asked questions

How much VRAM does self-hosted OpenVoice actually need? Is an 8GB card enough?

Enough, with room to spare. The V2 path is ~208MB for one MeloTTS language plus a 131MB converter — about 340MB of weights — and issue #48 in the repo reports roughly 1GiB of VRAM and 3GiB of system RAM at init. A single stream on an 8GB card is fine. Card selection is really about concurrency: every additional worker process costs another 0.3–0.5GB of CUDA context. On NexGPU, start on a Tesla V100 32GB at $0.188/GPU-hr, then move to an RTX 4090 at $0.540/GPU-hr when you scale concurrency. Billing is per second and compute billing stops when the instance stops.

What is the difference between OpenVoice V2 and V1, and which should I deploy?

V2 shipped in April 2024 as checkpoints_v2_0417.zip, replacing the base speaker with MeloTTS for native English, Spanish, French, Chinese, Japanese and Korean, with clearly better audio quality and an explicit MIT commercial licence. But it dropped V1's emotional style control — V1's English base speaker carries whispering, cheerful, sad and other style tags; V2 has none. So: V2 for multilingual quality, V1 if you need those emotion tags or you swap in your own expressive base speaker. Both weight sets together are under 1GB, so you can run them side by side on one NexGPU instance — an RTX 3090 24GB at $0.193/GPU-hr covers both comparisons in an hour.

Why does the cloned voice have the wrong accent and none of the reference speaker's emotion?

That is by design, not a bug. The official QA states it plainly: OpenVoice clones the tone color of the reference speaker and does NOT clone accent or emotion. Accent, pace and expression all come from the base speaker — in V2 that means one of MeloTTS's eleven speakers. Want British English, pick en-br; Indian English, en-india; Chinese-English code-switching, the Chinese base speaker. Genuine emotion transfer is outside what this architecture can do; you would need a different model. The fastest way to compare is to render all eleven embeddings from base_speakers/ses in one pass on a NexGPU instance — a few minutes of per-second billing.

My install keeps failing on numpy and librosa, and Silero VAD will not download. How do I fix it?

Both are long-standing. The first is frozen dependencies: requirements.txt pins numpy==1.22.0, librosa==0.9.1, gradio==3.48.0, faster-whisper==0.9.0 and whisper-timestamped==1.14.2, against an official Python 3.9 environment — installing on Python 3.11 with NumPy 2.x will not work. Use conda create -n openvoice python=3.9. The second is se_extractor pulling Silero VAD through torch.hub from GitHub; extract the zip into ~/.cache/torch/hub/snakers4_silero-vad_master manually. NexGPU has 2,000+ prebuilt images including PyTorch, so building a clean 3.9 env there beats fighting your system Python — and if you wreck it, destroying and relaunching costs a few cents.

Can I use OpenVoice commercially, and is there a watermark in the output?

The licensing is clean: V1 and V2 are both MIT, free for commercial and research use, confirmed alongside the V2 release in April 2024. The watermark is worth knowing about — wavmark==0.0.3 is a dependency, and the official demo_part3 passes encode_message = "@MyShell" into convert(), embedding that payload in the output audio. Decide what you want that message parameter to be before shipping. The harder compliance question is never the licence, it is whether you have the right to clone that person's voice, which is independent of the model. Running on NexGPU means reference and output audio stay on your own instance, reachable over SSH, Jupyter, REST API or CLI, with no third-party synthesis service in the path.

Is OpenVoice still worth deploying, or should I just use something like CosyVoice?

It depends what you are optimising for. The OpenVoice repo's last code push was 19 April 2025 and the V2 weights are still the April 2024 bundle — there is no V3, and this line has settled. But its advantages are hard to displace: under 350MB of weights, no autoregressive loop so latency is stable and predictable, MIT licensed, and an extremely high worker count per card. For batch-dubbing tens of thousands of clips offline it is an order of magnitude cheaper than LLM-style TTS. Conversely, for natural prosody and expression, Apache-2.0 CosyVoice (23k stars, still actively developed) is clearly stronger and considerably heavier. The rational move is to benchmark both: NexGPU has 1,175 verified rentable nodes across 75 GPU models, so run OpenVoice on a $0.188 V100 and CosyVoice on an A100 PCIE 80GB at $0.824/GPU-hr the same afternoon. Per-second billing, no minimum.

More in Speech synthesis

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.