Avatars / Lip Sync

Self-host MuseTalk: real-time lip sync that fits in 4GB of VRAM

The audio-driven lip-sync model from TMElyralab (Tencent Music's Lyra Lab). It inpaints faces in latent space instead of running a diffusion sampler, which is why it is one of the very few open models that genuinely holds 30fps. Rent a card and have your avatar talking in under an hour.

MuseTalk · self-hosted

MuseTalk does one narrow thing well: give it a video (or a still image) plus audio, and it repaints only the mouth-to-jaw region so the lips match the speech, leaving everything else untouched. It does not generate head motion, does not generate expression, and does not conjure a person out of nothing. Newcomers trip on this constantly — if you want a moving head, you need MuseV / MusePose upstream, or a real recorded driving video.

Architecturally it deliberately skips diffusion sampling. A frozen sd-vae-ft-mse compresses the 256×256 face region into latent space, a frozen whisper-tiny extracts audio features, and between them sits a UNet adapted from Stable Diffusion v1.4 (~0.85B parameters, 3.4GB fp32 checkpoint) that fuses audio into the image latent via cross-attention in a single step. No 20- or 50-step denoising loop. That is why the authors can claim "30fps+ on an NVIDIA Tesla V100", and why the memory footprint is absurdly small.

The current line is V1.5, released 28 March 2025: it adds perceptual loss, GAN loss and sync loss on top of V1.0, with a two-stage training strategy and spatio-temporal sampling (Informative Frame Sampling plus Dynamic Margin Sampling), giving clearly better clarity, identity consistency and lip-speech alignment. The repository's last commit lands in September 2025 and there is no 2.0 — V1.5 is the version to deploy today.

01 —

Versions and weights: what to actually download

download_weights.sh pulls the whole set — don't grab the UNet alone

Version	Parameters	VRAM	Context	Notes
MuseTalk V1.5 (models/musetalkV15/unet.pth)	UNet ≈0.85B, 3.40GB checkpoint	~4GB in fp16; ~3.6GB observed in the field	256×256 face region / 25fps	The version to use. Perceptual + GAN + sync loss, two-stage training. `--version v15` pairs with the `--extra_margin 10` and `--parsing_mode jaw` defaults.
MuseTalk V1.0 (models/musetalk/pytorch_model.bin)	Also a 3.40GB checkpoint	~4GB in fp16	256×256 / 25fps	The April 2024 original. The only reason to keep it is `--bbox_shift`, which only takes effect on V1.0 — positive values open the mouth wider, negative values close it. Run once to see the printed valid range, then pick a value.
Real-time mode (scripts/realtime_inference.py)	batch_size defaults to 20 (offline script uses 8)	Scales linearly with batch on top of the V1.5 weights	Official 30fps+ on a Tesla V100	Run preparation once to cache coords.pkl, latents.pt, mask/ and full_imgs/ under results/v15/avatars/<avatar_id>/. After that each audio clip only walks Whisper → UNet → VAE decode → blend.
Training Stage 1 (latent inpainting)	8×H20, batch 32, grad accumulation 1	≈74GB per GPU	256×256	Learns the latent inpainting task itself. An 80GB card just fits — this stage is not the bottleneck.
Training Stage 2 (GAN + sync refinement)	8×H20, batch 2, grad accumulation 8	≈85GB per GPU	256×256	Adds the discriminator and SyncNet; batch drops to 2 yet memory goes up. 80GB cards hit the ceiling here — you need the 141GB tier.
Companion weights (sd-vae-ft-mse / whisper-tiny / DWPose / face-parse-bisent)	VAE + audio encoder + landmarks + face parsing	Under 1GB combined	—	DWPose ships as dw-ll_ucoco_384.pth; face parsing needs 79999_iter.pth plus resnet18-5c106cde.pth. Training additionally pulls latentsync_syncnet.pt from ByteDance/LatentSync.

02 —

Which NexGPU card to rent

MuseTalk is compute-bound, not memory-bound — the selection logic is nothing like serving an LLM

Quality checks and offline batch dubbing (dozens to hundreds of clips)
Tesla V100 32GB$0.188/GPU-hr
The official "30fps+" figure was measured on exactly this card, so there is no guesswork — and it is the cheapest tier on the platform.
Live avatar streaming (realtime mode + TTS + NVENC push)
RTX 4090 24GB$0.540/GPU-hr
Ada's fp16 throughput saturates batch_size 20 while leaving the encoder free for the outbound stream, so you don't need a second box.
Chaining GFPGAN / CodeFormer to upscale 256×256 to 1080p, or running several streams
RTX A6000 48GB$0.817/GPU-hr
MuseTalk itself takes only a few GB; 48GB lets the super-resolution model and multiple avatar caches stay resident instead of swapping.
Two-stage training or fine-tuning on your own footage
H200 141GB$6.660/GPU-hr
Stage 2 measures ≈85GB per GPU, which puts 80GB A100s and H100s just under the line. 141GB runs it without touching the hyperparameters.

03 —

Empty instance to finished clip, in four steps

Start from a PyTorch prebuilt image on NexGPU; the whole sequence takes about half an hour

01
Build the env, including the mm-stack
Pin Python 3.10. The step that actually stops people is mmcv/mmdet/mmpose, which DWPose depends on: install those exact versions through openmim, because a plain pip install will usually fail to compile.
```
conda create -n MuseTalk python==3.10 -y && conda activate MuseTalk && pip install -r requirements.txt && mim install mmengine "mmcv==2.0.1" "mmdet==3.1.0" "mmpose==1.1.0"
```
02
Pull every weight in one shot
The script fetches musetalkV15/unet.pth (3.4GB), sd-vae-ft-mse, whisper-tiny, DWPose's dw-ll_ucoco_384.pth, and face-parse-bisent's 79999_iter.pth plus resnet18. Miss any one and it blows up at face detection or blending.
```
sh download_weights.sh
```
03
Run offline inference to confirm quality
Point configs/inference/test.yaml at your driving video and audio. Convert the driving video to 25fps first — the model was trained at 25fps and a mismatched frame rate drifts the whole lip track. Adding --use_float16 roughly halves memory.
```
python -m scripts.inference --inference_config configs/inference/test.yaml --result_dir results/test --unet_model_path models/musetalkV15/unet.pth --unet_config models/musetalkV15/musetalk.json --version v15 --use_float16
```
04
Switch to real-time and bake a reusable avatar
Set preparation to true in configs/inference/realtime.yaml, run once, then flip it back to false — after that, swapping audio responds in seconds. If you are chasing frame rate, always pass --skip_save_images, or the bottleneck becomes writing PNGs to disk.
```
sh inference.sh v1.5 realtime
```

What it actually costs

Take 100 clips of 60-second talking head. At 25fps that is 1,500 frames each; at the official Tesla V100 rate of 30fps+, the UNet portion runs about 50 seconds per clip. Add DWPose detection, VAE encode/decode and the ffmpeg mux and call it a conservative 2 minutes per clip. 100 clips = 200 minutes ≈ 3.34 hours. Tesla V100 32GB is $0.188/GPU-hr, so 3.34 × 0.188 ≈ $0.63. Give the setup — environment plus the 3.4GB unet.pth and the companion weights — 30 minutes: 0.5 × 0.188 ≈ $0.09. The whole batch lands under $0.75. Shipping roughly 10GB of 1080p output costs 10 × $0.0081 ≈ $0.08. For a live avatar instead: RTX 4090 24GB at $0.540/GPU-hr, eight hours a day, is 8 × 0.540 = $4.32 per day, or ≈ $95 over 22 broadcast days. Billing is metered per second and priced per hour, and compute billing stops the moment the instance stops. Only the weights and avatar cache left on disk keep accruing (model plus cache is about 12GB, at $0.414/GB-month ≈ $4.97/month), and that ends when you destroy the storage. No minimum term, no setup fee, no quota request.

04 —

FAQ

How much VRAM does MuseTalk really need to self-host?

Far less than you'd guess. The official README's lowest tested configuration is a 4GB RTX 3050 Ti Laptop GPU in fp16, where an 8-second video takes about 5 minutes; a community field report on an RTX 5060 Ti measured roughly 3.6GB used by the models. The fp32 checkpoint alone is 3.40GB, so without fp16 an 8GB card is the safer floor. Throughput, not memory, decides your experience — a 4GB card produces output but is nowhere near real time. To match the official 30fps+ directly, rent a Tesla V100 32GB on NexGPU at $0.188/GPU-hr.

Is there a MuseTalk 2.0? Which version should I deploy?

There is no 2.0. The current line is V1.5, released 28 March 2025, and the repository's last commit is from September 2025 (normalising all audio to 16kHz WAV and fixing the Windows download script). Unless you specifically need `--bbox_shift`, which only works on V1.0, go with V1.5. Want to A/B the two? NexGPU bills per second — spin up two instances, run the comparison, stop them, and it costs pocket change.

The output is only 256×256 and the face looks soft. How do I fix it?

That is an architectural ceiling, not a misconfiguration — the model is trained on a 256×256 face region. The authors state the limitations plainly: details like a moustache, lip shape and lip colour are not well preserved, and per-frame generation introduces slight jitter. The standard remedy is chaining GFPGAN or CodeFormer for super-resolution, plus tuning `--extra_margin`, `--left_cheek_width` and `--right_cheek_width` (both default to 90) to clean up the blend boundary. To keep the upscaler and MuseTalk resident on one card, take an RTX A6000 48GB at $0.817/GPU-hr.

Can real-time mode actually hit real time?

Yes, if you avoid the trap. You must run preparation once so latents.pt, coords.pkl and the masks are cached; after that each clip is just Whisper feature extraction, UNet inference, VAE decode and blending. The repo's issue tracker keeps asking whether anyone has genuinely achieved real time, and in most cases the answer is a missing `--skip_save_images` — the frame budget is being spent writing PNGs to disk. For headroom to encode and stream at the same time, RTX 4090 24GB at $0.540/GPU-hr on NexGPU is the easy answer.

MuseTalk throws "no kernel image" on RTX 50-series (Blackwell). What now?

sm_120 needs PyTorch built against CUDA 12.8 or newer; 2.6.0+cu124 will always fail. The second trap is that mmcv/mmpose have no prebuilt wheels for Python 3.12, and the community workaround replaces DWPose with mediapipe's 478-point face mesh, remapping the nose-bridge points back to the indices MuseTalk expects. If you'd rather not fight it, sidestep the whole thing: RTX 4090 24GB at $0.540 and Tesla V100 32GB at $0.188 on NexGPU just work. If you do need Blackwell, RTX 5090 32GB is on the list at $0.723/GPU-hr.

Can I use MuseTalk commercially? What's the licence?

The code is MIT and the weights are published on Hugging Face under creativeml-openrail-m, with the authors' disclaimer stating the trained model is available for any purpose, including commercial use. Two caveats: the sample data in the repo is for non-commercial research only, and dependencies like whisper, sd-vae-ft-mse and DWPose carry their own licences that you must honour separately. As for where to run it, NexGPU spans 51 countries and regions with 1,175 verified rentable nodes, so you choose the jurisdiction, and support is bilingual over Telegram with no ticket queue.

More in Digital humans and face animation

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.