Skip to main content
NexGPU

Speech Synthesis · TTS

Self-hosting Tortoise TTS: 24kHz voice cloning that starts at 4GB of VRAM

Five models chained into one pipeline, 4.6GB of weights entirely on your own disk, and a voice pinned down by a handful of reference clips. Slow is literally its name, but the v3 hifidecoder pushed RTF down to 0.25-0.3.

Sign up freeView pricing

The name is James Betker's own joke about the first release: a medium-length sentence took two minutes on a K80. What that bought was prosody nobody else had at the time, because Tortoise is not one end-to-end model but a relay of five. A GPT-2-style autoregressive decoder (autoregressive.pth, 1.72GB in fp32) emits dozens to hundreds of candidate mel-token sequences, CLVP2 (976MB) — a CLIP-like text-to-speech scorer — reranks them and keeps the one that best matches the text, a diffusion decoder (1.17GB) turns that into a mel spectrogram, and the UnivNet vocoder (391MB) renders 24kHz audio. Reference clips are loaded at 22.05kHz for conditioning; output is always 24kHz.

Betker built the whole thing alone on his own hardware: eight RTX 3090s over several months, roughly 50k hours of speech weighted heavily toward audiobooks, trained with DLAS, a framework he also wrote. Code lives at github.com/neonbjb/tortoise-tts under Apache 2.0, weights at jbetker/tortoise-tts-v2 on HuggingFace. Be clear about the project's state: upstream has been quiet for a long while. PyPI carries exactly one release, tortoise-tts 3.0.0 from October 2023, and it is the important one — it shipped the hifidecoder (71MB), which replaces both the diffusion decoder and UnivNet and generates waveforms straight from autoregressive latents. The README's number for that path is "0.25-0.3 RTF on 4GB vram", with sub-500ms first-chunk latency through tts_stream. The ecosystem has contracted too: JarodMica/ai-voice-cloning, which carried mrq's training fork, was archived in June 2025, and 152334H/tortoise-tts-fast is flagged maintenance only.

So why this page? Because two things about Tortoise were never quite replaced. First, the reading voice that comes out of the full diffusion path — long-sentence pacing, the rise and fall of narration, prose and poetry — which people still keep it around specifically for. Second, Apache 2.0 with fully offline weights: nothing phones home, nothing is metered by a vendor, and there is no licensing conversation to have before you deploy it privately. This page covers what it actually costs in VRAM, how the four presets differ, how to get a dependency set frozen in 2023 to install on a 2026 machine, and which NexGPU card to pick — starting at $0.193/GPU-hour for an RTX 3090 24GB. Every size and VRAM figure here comes from the actual byte counts of the published checkpoints and from upstream's own README, so you can check them yourself.

01 —

Five models, two inference paths

There is only one weight release. What you actually choose is which decode path to take and which optional modules to load.

VersionParametersVRAMContextNotes
api_fast.py + hifidecoder (the v3.0.0 fast path)autoregressive 1.72GB + hifidecoder 71MB~1.8GB of fp32 weights; ~0.9GB with half=True; upstream quotes 4GB VRAMmax_mel_tokens 500 (roughly one sentence) / 24kHz outputSkips the diffusion decoder and UnivNet entirely, going straight from AR latents to waveform. The only path with tts_stream chunked streaming (stream_chunk_size defaults to 40, overlap_wav_len 1024), and the source of that 0.25-0.3 RTF figure. If you need low latency, this is the only option.
api.py full diffusion path (the classic v2 pipeline)AR 1.72GB + diffusion 1.17GB + CLVP2 976MB + UnivNet 391MB~4.3GB of resident fp32 weights, ~2.2GB with half=True; peak set by AR candidate batchtts() defaults to num_autoregressive_samples=512, diffusion_iterations=100The path that makes Tortoise sound like Tortoise. All four presets differ on just two numbers: ultra_fast is 16 candidates / 30 diffusion steps, fast is 96/80, standard is 256/200, high_quality is 256/400. Cut candidates first to save time; cut diffusion steps only when you're willing to spend quality.
CVVP voice reranking (off by default)cvvp.pth 151MB+~0.15GBcvvp_amount defaults to 0.0A second scorer. CLVP2 asks "does this match the text?", CVVP asks "does this match the speaker?". When candidates come back with the right words in the wrong voice, dial cvvp_amount up to 0.1-0.3, at the cost of another loaded model and a slower rerank.
RLG random voice (rlg_auto + rlg_diffuser)25MB + 101MB+~0.13GBvoice='random'Samples a speaker who does not exist when you supply no reference audio. The native answer for anonymous narration, placeholder VO, or any compliance situation where you would rather not touch a real person's voiceprint at all.
tortoise-detect classifierclassifier.pth 61MB~0.1GBCalled standalone; not part of synthesisThe provenance tool Betker published alongside the model: it judges whether a clip was generated by Tortoise. Handy in a private deployment for output watermarking and internal audit without contracting a third-party detection service.
tortoise-tts-fast (152334H community fork)Reuses the official weights, no extra downloadSame as the full path; CPU offload available to trade speed for VRAMDDIM / DPM-Solver samplersThe community's speed answer from before upstream shipped the hifidecoder, claiming 5-10x and up to roughly 20x via kv_cache, fp16 and swapped diffusion samplers. The repo is marked maintenance only, but if you want the diffusion sound quality *and* more speed, it is still the only off-the-shelf option.

02 —

Which NexGPU card to pick

Upstream's floor is 4GB, so the question with Tortoise is never "will it fit" — it is "how fast does AR sampling run".

  • Auditioning voices, ultra_fast preset, hifidecoder streaming

    RTX 3090 24GB$0.193/GPU-hour

    Upstream says 4GB is enough, so 24GB is wild overkill — but it is also the cheapest 24GB card on our list, and with fp16 plus kv_cache you still won't fill it, so there is no reason to pay more for a smaller card.

  • standard / high_quality presets: 256 candidates and 200-400 diffusion steps

    RTX 4090 24GB$0.540/GPU-hour

    This tier is pure compute — candidate count and diffusion steps both jump by more than an order of magnitude and AR sampling becomes the hard bottleneck, so 4090 fp16 throughput usually wins on total bill by finishing sooner.

  • Bulk offline audiobook synthesis with per-sentence resampling

    RTX 5090 32GB$0.723/GPU-hour

    The extra 8GB goes straight into a larger autoregressive_batch_size, so you place more bets per pass and let CLVP2 sort them in one rerank instead of babysitting read.py's --regenerate loop.

  • Fine-tuning the autoregressive backbone with DLAS for a bespoke voice

    RTX A6000 48GB$0.817/GPU-hour

    Training holds weights, gradients and optimizer state at once, and 48GB is the lowest tier that runs without code surgery — though note the A100 PCIE 80GB is $0.824, so $0.007 more per hour doubles your VRAM. Compare both before you launch.

03 —

From bare instance to first audio

The hard part of installing Tortoise isn't the model, it's the frozen requirements.txt. Isolate the environment first and the rest is smooth.

  1. 01

    Launch an instance on a PyTorch image and pull the code

    Pick an RTX 3090 24GB at console.nexgpu.net and one of the prebuilt PyTorch images (there are 2,000+ to choose from), then SSH in and clone. No CUDA install, no quota request.

    git clone https://github.com/neonbjb/tortoise-tts.git && cd tortoise-tts
  2. 02

    Build an isolated conda environment and dodge the dependency mines

    This is the only step that genuinely trips people up. The official requirements.txt pins transformers==4.31.0, deepspeed==0.8.3, pydantic==1.9.1, librosa==0.9.1 and tornado==4.2 — a set of 2023-era versions that will almost certainly collide with anything else on the machine. A dedicated Python 3.9 environment is the path of least resistance. Do not reuse base to save time.

    conda create -n tortoise python=3.9 -y && conda activate tortoise && pip install "transformers==4.31.0" && pip install -e .
  3. 03

    Generate your first line; weights download themselves

    The first run pulls the full weight set from HuggingFace — eight files totalling roughly 4.6GB, cached under ~/.cache. --voice takes any of the thirty voices bundled in tortoise/voices/ (tom, emma, freeman, geralt, applejack and friends), or random to have RLG invent a speaker on the spot. To clone your own voice, make a new directory under voices/ and drop in a few clean multi-second wavs. No training required.

    python tortoise/do_tts.py --text "The quick brown fox jumps over the lazy dog." --voice tom --preset fast
  4. 04

    Switch to the streaming API with every optimization on

    Once you hear sound, move to api_fast: kv_cache stops AR sampling from recomputing prefixes, half runs fp16, and use_deepspeed engages the DeepSpeed inference kernels. Those three flags are the precondition for 0.25-0.3 RTF and sub-500ms first chunks. To serve it, the repo ships tortoise/socket_server.py, which listens on port 5000.

    python -c "from tortoise.api_fast import TextToSpeech; tts = TextToSpeech(use_deepspeed=True, kv_cache=True, half=True)"

What an audiobook actually costs

Start from the one number upstream commits to: 0.25-0.3 RTF on the hifidecoder path. Take the upper bound of 0.3, meaning one hour of finished speech occupies 0.3 GPU-hours. On an RTX 3090 24GB at $0.193/GPU-hour, that hour of audio costs 0.3 x 0.193 = $0.058. A nine-hour audiobook is 2.7 GPU-hours, so 2.7 x 0.193 = $0.52. Add two hours for the first weight pull, the conda environment and getting do_tts.py to speak: 2 x 0.193 = $0.386. Total about $0.91 — less than a coffee. If what you want is the narration quality Tortoise is actually known for, you're on the full diffusion path. The standard preset means 256 candidates and 200 diffusion steps, more than an order of magnitude slower than the hifidecoder, and the exact multiplier depends on sentence length and batch size, so measure thirty seconds of your own text before extrapolating. Assuming a rough 10x (RTF 3), that same nine-hour audiobook needs 27 GPU-hours; on an RTX 4090 24GB at $0.540/GPU-hour that's $14.58, or one night of unattended runtime. Storage is billed separately: 4.6GB of weights at the $0.414/GB-month median works out to $1.90/month. Mind the billing boundary — compute stops the moment the instance stops, but storage keeps running until you destroy the volume. So stop the instance after each batch and keep the weights around; that beats re-downloading 4.6GB every session. Everything is metered per second and priced per hour, with no minimum, no setup fee and no quota request.

04 —

Frequently asked questions

How much VRAM does Tortoise TTS actually need to self-host?

Upstream's README states "0.25-0.3 RTF on 4GB vram", which refers to the v3 hifidecoder path: only autoregressive.pth (1.72GB) and hifidecoder.pth (71MB) stay resident, and with half=True the weights come in under 1GB. On the full diffusion path, AR 1.72GB + diffusion 1.17GB + CLVP2 976MB + UnivNet 391MB adds up to about 4.3GB resident, roughly 2.2GB in fp16. But peak VRAM isn't set by weights — it's set by num_autoregressive_samples, and the standard preset places 256 bets at once. That's where you OOM. Practically: an 8GB card works, but there's no reason to squeeze. An RTX 3090 24GB on NexGPU is $0.193/GPU-hour, so you can max the candidate batch and stop thinking about it.

Why is Tortoise TTS so slow, and can I speed it up?

It's slow by design: it autoregressively samples tens to hundreds of candidates, reranks each one with CLVP2, then runs hundreds of diffusion steps — three of those four stages are serial. Three ways to speed it up. One, use the hifidecoder fast path (api_fast.py), which drops diffusion and the vocoder; that's upstream's own answer and the origin of the 0.25-0.3 RTF number. Two, turn on kv_cache=True, half=True and use_deepspeed=True together. Three, step down the preset — high_quality (256/400) to fast (96/80) to ultra_fast (16/30); candidate count is the most valuable knob you have. The community's tortoise-tts-fast swaps in DDIM/DPM-Solver samplers and claims 5-10x. Honestly, the easiest speedup is a better card: the same code on an RTX 4090 at $0.540/GPU-hour finishes sooner, and a shorter run often means a smaller bill.

Does Tortoise TTS support languages other than English?

The official weights do not. Training was roughly 50k hours of predominantly English audiobooks, and the tokenizer and phoneme coverage are English-only by design — feed it Chinese or Japanese and you get an accented mangling or nothing usable. Community forks (mrq's lineage, later maintained by JarodMica) added multilingual support, but that repo was archived in June 2025 and its maintainer's own README points people toward XTTS, GPT-SoVITS, F5-TTS, StyleTTS2 or Zonos instead. Treat this page as an English-narration solution. The good news is that switching models on NexGPU just means switching images — the instance and the billing work exactly the same way.

Installation fails with transformers, pydantic and DeepSpeed version conflicts. What now?

This is the single most real pain point in running Tortoise on a modern machine. requirements.txt pins transformers==4.31.0, deepspeed==0.8.3, pydantic==1.9.1, einops==0.4.1, librosa==0.9.1, tornado==4.2 and nbconvert==5.3.1 — all 2023 versions — and the official conda recipe calls for Python 3.9 with PyTorch cu117. pydantic 1.x and transformers 4.31 essentially cannot coexist with a current stack, and force-upgrading transformers breaks on GPT2 API changes. There is one correct answer: total isolation, via a dedicated conda environment or, better, a dedicated instance. Never touch system Python. Also note DeepSpeed is disabled on Apple Silicon and frequently fails to build on Windows — Linux is where this is easy. NexGPU instances are clean Linux boxes, so if you wreck one, stop it and start another; per-second billing means a botched install costs pennies.

Has Tortoise been superseded by XTTS and F5-TTS? Is it still worth deploying?

On speed and language coverage, yes — Coqui's XTTS grew directly out of Tortoise's architectural ideas, and F5-TTS, GPT-SoVITS and StyleTTS2 all beat it comfortably, while upstream Tortoise sits at 3.0.0 from October 2023. Two reasons people still deploy it anyway. First, the English narration voice from the full diffusion path: the pacing on long sentences, the intonation on prose and poetry, which some teams keep it around exclusively for. Second, Apache 2.0 with a complete offline weight set, plus the tortoise-detect classifier the author shipped alongside it — that combination makes a fully air-gapped private deployment nearly frictionless on licensing and compliance. If either of those matters to you, spin up an RTX 3090 at $0.193/GPU-hour, generate a few passages, and listen. If it isn't for you, stop the instance; there's no minimum rental period.

I want to fine-tune my own voice. How big a GPU do I need?

First, most people don't need to. Tortoise's zero-shot cloning is usually enough: create a folder under tortoise/voices/, drop in a few clean multi-second wavs, point --voice at it, and you've trained nothing. If you genuinely want to fine-tune the autoregressive backbone, that's done with DLAS, the author's own training framework (the GUI wrapper around it came from mrq and was later taken over by JarodMica; that repo is archived, but the code still runs). Training holds weights, gradients and optimizer state simultaneously, so VRAM demand jumps a tier: NexGPU's RTX A6000 48GB at $0.817/GPU-hour is the lowest option that runs without code changes, but the A100 PCIE 80GB is $0.824 — $0.007 more per hour for double the VRAM, which is usually the better buy. Across our network there are 1,175 verified rentable nodes and 2,498 GPUs, up to 14 GPUs per node and 2,152GB max node VRAM, so scaling out is available if you need it.

More in Speech synthesis

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.