Skip to main content
NexGPU

Digital Human / Lip Sync

Self-hosting Wav2Lip: what a 96x96 mouth actually costs in VRAM

Short version: Wav2Lip only generates a 96x96 mouth region. The VRAM ceiling comes from S3FD face detection in front of it, not the model, and a 6GB card clears it. What actually burns GPU time is the upscaler you bolt on afterwards. NexGPU starts at $0.193/GPU-hour on an RTX 3090 24GB, metered per second.

Sign up freeView pricing

Wav2Lip comes from the ACM Multimedia 2020 paper A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild, a collaboration between IIIT Hyderabad and the University of Bath. The repo lives at github.com/Rudrabha/Wav2Lip with 13.2k stars, 2.8k forks and 113 commits on master. The insight was counterintuitive at the time: instead of building a fancier generator, train a SyncNet expert discriminator on LRS2 that does nothing but judge whether audio and lips agree, freeze it, and use it to punish the generator. The syncnet_wt=0.0 line in hparams.py is the switch for that curriculum — let the generator learn faces first, turn on the sync penalty once reconstruction loss has settled. The conditioning window is tiny: 16kHz audio, 80 mel bins, hop_size=200 means 80 mel frames per second, and inference.py fixes mel_step_size=16, exactly 0.2 seconds, which lines up with 5 video frames at 25fps.

Clone it today and the first wall you hit is not VRAM, it is dependencies. requirements.txt pins torch==1.1.0, torchvision==0.3.0, librosa==0.7.0, numpy==1.17.1 and numba==0.48, and the README asks for Python 3.6. None of that installs in a modern CUDA image: numba 0.48 will not build against Python 3.10; move to a current librosa and audio.py throws a TypeError because librosa.filters.mel(sr, n_fft, ...) became keyword-only; numpy 1.24 deleted np.float, which breaks the face detection path with an AttributeError; and torch 2.6 flipped torch.load to weights_only=True, so loading wav2lip_gan.pth fails again. The good news is that the model code itself is fine — patch those four spots and it runs faster on torch 2.x than it ever did on 1.1.0. If you would rather not, the community already solved it: Easy-Wav2Lip pins Python 3.10.11 and CUDA 12.2, mowshon/lipsync is a pip install lipsync away from LipSync(model='wav2lip', img_size=96, device='cuda'), and instant-high's wav2lip-onnx-HQ ports the whole pipeline to ONNX with four face enhancers, tested by the author on an RTX 3060 6GB.

One thing to settle before you build anything on it: the licence covers personal, research and non-commercial use only. The README states that any form of commercial use is strictly prohibited and points commercial users at Sync Labs ([email protected], [email protected]). So if this is a product rather than an experiment, the successors are what you should be evaluating. ByteDance's LatentSync is Apache-2.0 and version 1.6 trains at 512x512, with official inference VRAM listed at 18GB (the 256x256 v1.5 needed 8GB). TMElyralab's MuseTalk 1.5 is MIT, explicitly permits commercial use of the trained weights, works on a 256x256 face region and hits 30fps+ real-time inference on a Tesla V100. Wav2Lip still holds one seat nobody has taken: it is the lightest, fastest and most forgiving of bad footage, and 96x96 buys you a VRAM footprint you can basically ignore. On NexGPU all of that is billed by the second, from $0.193/GPU-hour on an RTX 3090 to $0.824/GPU-hour on an A100 PCIE 80GB, so proving out a pipeline before you upgrade costs minutes of compute.

01 —

Which Wav2Lip weights you should actually run

The project ships two checkpoints; the real choice happens among the forks and the successors

VersionParametersVRAMContextNotes
wav2lip.pth (official baseline)Convolutional encoder-decoder GAN; audio branch takes 80 mel bins, face branch takes a 5-frame windowUnder 4GB for inference; the peak is set by S3FD at face_det_batch_size=16, not by the model96x96 generated face; mel_step_size=16 maps to 0.2s / 5 frames at 25fpsThe README calls this highly accurate lip-sync. It is the most synchronised of the two, and the blurriest. Use it for sync metrics, evaluation baselines and data labelling.
wav2lip_gan.pth (official visual-quality variant)Same generator plus a visual quality discriminator (disc_wt=0.07 in hparams)Identical footprint, under 4GB for inferenceAlso 96x96 outputThe README describes it as slightly inferior lip-sync with better visual quality. This is what most people producing finished video should use. Both checkpoints are manual Google Drive downloads, and s3fd.pth has to be placed at face_detection/detection/sfd/ separately.
Easy-Wav2Lip (anothermartz fork)Original weights plus a feathered mouth mask and optional GFPGANFast mode matches the original; Enhanced mode's footprint and runtime are set by GFPGANThree quality tiers: Fast / Improved / EnhancedPins Python 3.10.11 and CUDA 12.2 so you skip the dependency archaeology entirely. The published comparison on a Colab T4 with a 9-second 720p60 clip: 6m53s for the original, 56s for this, and 25s on a repeat run of the same video once face tracking is cached.
wav2lip-onnx-HQ (instant-high fork)Full pipeline exported to ONNX with four face enhancers built inAuthor reports it running on an RTX 3060 6GB; also runs CPU-only96x96 backbone with the enhancer restoring native resolutionThe pick when you need Wav2Lip on a small card, or inside a serving stack that has no PyTorch. Fast startup and few dependencies, at the cost of being awkward to modify.
LatentSync 1.6 (ByteDance, Apache-2.0)Audio-conditioned latent diffusion with no intermediate motion representation18GB inference (the 256x256 v1.5 needed 8GB); training stage 1 23GB, stage 2 30GB, or 20GB in the memory-efficient path512x512 output; 512 training needs 30GB for stage 1 and 55GB for stage 2It openly borrows code from Wav2Lip but takes a completely different route. Sharpness is a clear step up over 1.5, which itself was tuned for Chinese-language video. If you need commercial rights and HD, start here.
MuseTalk 1.5 (TMElyralab, MIT)Real-time lip driving with weights explicitly cleared for commercial useRuns in fp16 on a 4GB RTX 3050 Ti (about 5 minutes for an 8-second clip); training needs ~74GB per GPU for stage 1 and ~85GB for stage 2256x256 face region; 30fps+ real-time inference on a Tesla V100The other main line beyond Wav2Lip if you are building a live avatar or a low-latency interactive agent. Chain GFPGAN behind it when you need more resolution.

02 —

Which GPU to rent for Wav2Lip

Decide first whether you are running the 96x96 backbone or the upscaler that actually eats the card

  • Get the stock 96x96 pipeline working and batch out a few dozen talking-head clips

    RTX 3090 24GB$0.193/GPU-hour

    The Wav2Lip backbone barely touches VRAM, so all 24GB goes to letting S3FD run at the default face_det_batch_size=16 on full 1080p frames instead of forcing you into --resize_factor.

  • Wav2Lip plus GFPGAN or CodeFormer upscaling for delivery-grade output

    RTX 4090 24GB$0.540/GPU-hour

    The upscaler runs frame by frame and is the real time sink; the 4090's fp16 throughput turns Easy-Wav2Lip's Enhanced mode from an overnight job into a coffee break.

  • Retrain the SyncNet expert discriminator and the Wav2Lip generator yourself

    RTX A6000 48GB$0.817/GPU-hour

    syncnet_batch_size=64 in hparams is far heavier than the generator's batch_size=16, and 48GB lets you run the stock configuration without touching a single hyperparameter.

  • Switch to LatentSync 1.6 for 512x512 output, or run its second training stage

    A100 PCIE 80GB$0.824/GPU-hour

    Official figures are 18GB for 1.6 inference and 55GB for 512-resolution stage 2 training, so one 80GB card covers both — and it costs less than the A100 SXM4.

03 —

Getting Wav2Lip running on NexGPU

Four steps, and step two is where everyone gets stuck

  1. 01

    Launch a PyTorch image and pull the repo and the weights

    At console.nexgpu.net pick any 24GB card and boot the PyTorch entry from the 2,000+ prebuilt images. The repo ships no weights: wav2lip.pth and wav2lip_gan.pth come from the Google Drive links in the README, and the s3fd.pth face detector has to land at face_detection/detection/sfd/. One character wrong in that path and you get a FileNotFoundError.

    git clone https://github.com/Rudrabha/Wav2Lip && mkdir -p Wav2Lip/face_detection/detection/sfd
  2. 02

    Ignore requirements.txt and patch four places

    requirements.txt pins torch==1.1.0, librosa==0.7.0, numba==0.48 and numpy==1.17.1, which cannot install on modern Python. Install current versions instead, then fix these four: convert every librosa.filters.mel call in audio.py to keyword arguments (sr=, n_fft=, n_mels=); replace leftover np.float with float; pass weights_only=False to torch.load; and swap librosa's removed .output.write_wav for soundfile. Once patched, the model runs cleanly on torch 2.x and faster than it did on 1.1.0.

    pip install torch torchvision librosa soundfile opencv-python numpy tqdm ffmpeg-python
  3. 03

    Run your first inference, tuning --pads and --resize_factor first

    The default pads=[0,10,0,0] regularly clips the chin; talking-head footage usually wants the bottom pad around 20. If S3FD runs out of memory on 1080p input, the script halves face_det_batch_size on its own and prints Recovering from OOM error; only when it reaches 1 does it raise Image too big to run face detection on GPU, and that is when --resize_factor 2 earns its keep. The README is explicit that the model does better on lower-resolution faces, and 720p output often looks more natural than 1080p. Add --nosmooth the moment you see the mouth drifting out of place.

    python inference.py --checkpoint_path checkpoints/wav2lip_gan.pth --face input.mp4 --audio speech.wav --pads 0 20 0 0 --nosmooth
  4. 04

    Bolt on an upscaler to hide the 96x96 patch

    The stock pipeline pastes a 96x96 mouth straight back into the original frame, and at 1080p the seam and softness are unmistakable. Two routes: Easy-Wav2Lip's Improved mode, which uses a feathered mask so only the mouth is replaced and the rest of the face keeps native resolution, or Enhanced mode, which adds per-frame GFPGAN restoration on top. Download your output and stop the instance — NexGPU halts compute billing the moment an instance stops, and only storage keeps accruing until you destroy it.

    python Easy-Wav2Lip.py  # set quality to Improved or Enhanced in config.ini

A real number: 100 clips of 30-second talking head

Anchor on the benchmark Easy-Wav2Lip publishes: 56 seconds for a 9-second 720p60 clip on a Colab T4. A 30-second clip is roughly 3.3x that, so call it 3 minutes per clip on a T4. Move to an RTX 4090 24GB ($0.540/GPU-hour) and, conservatively assuming only a 3x speedup, that is about 1 minute per clip. 100 clips is 100 minutes, or 1.67 hours: 1.67 x $0.540 = $0.90. Add 20 minutes for setup, pulling weights and dialling in --pads: 0.33 x $0.540 = $0.18. Total compute, about $1.08. If you skip upscaling and run the stock pipeline on an RTX 3090 24GB ($0.193/GPU-hour), the same wall time costs 1.67 x $0.193 = $0.32. Storage is separate: 20GB of source footage at the $0.414/GB-month median is $8.28 for a full month, or roughly $0.28 for a single day; 2GB of finished video egressing at the $0.0081/GB median is about $0.02. So the one-off compute bill for the batch lands a shade over a dollar — provided you remember to stop the instance. Billing is metered per second and priced per hour, with no minimum, no setup fee and no quota request.

04 —

Frequently asked questions

How much VRAM does Wav2Lip actually need to run locally?

Almost none for the model itself. inference.py hardcodes img_size = 96, so the generator only ever processes a 96x96 face crop, and even the default wav2lip_batch_size of 128 stays small. The real peak comes from S3FD face detection in front of it, which runs at the source frame resolution with face_det_batch_size=16 by default — that is where 1080p footage OOMs. The script self-heals by halving the batch and printing Recovering from OOM error, and only raises Image too big to run face detection on GPU once it reaches 1. instant-high's ONNX fork is documented working on an RTX 3060 6GB, which is a fair lower bound. On NexGPU the simplest answer is an RTX 3090 24GB at $0.193/GPU-hour, with enough headroom that you never have to trade away resolution.

Can I use Wav2Lip commercially? What is the legal exposure?

No. The README is blunt: the repository may only be used for personal, research and non-commercial purposes, any form of commercial use is strictly prohibited, and commercial licensing goes through Sync Labs ([email protected], [email protected]). That restriction travels to the community forks too, since Easy-Wav2Lip and wav2lip-onnx-HQ ship the same checkpoints. For a commercial avatar product the honest alternatives are MuseTalk 1.5 (MIT, weights explicitly cleared for commercial use) or LatentSync 1.6 (Apache-2.0). NexGPU has the card for either: an RTX 4090 24GB is more than enough for MuseTalk, while LatentSync 1.6's 18GB inference requirement fits comfortably on an A100 PCIE 80GB at $0.824/GPU-hour.

Why is the mouth blurry, and why is there a visible square around it?

Because the output really is 96x96. It gets scaled and pasted back into the original frame, so the higher your source resolution the more obvious the patch and the sharpness gap become — which is exactly why the README says the model performs better on lower-resolution faces, and why many people find 720p output looks more natural than 1080p. Three fixes: use --resize_factor to bring the whole frame down so the gap is less visible; use Easy-Wav2Lip's Improved mode, which feathers a mask so only the mouth is replaced; or Enhanced mode, which layers per-frame GFPGAN restoration on top. The last two cost GPU time, and NexGPU's RTX 4090 24GB at $0.540/GPU-hour is metered per second, so you stop paying the moment the render finishes.

requirements.txt will not install and I get an np.float error. How do I fix it?

You are looking at a 2020 environment frozen in place. requirements.txt pins torch==1.1.0, librosa==0.7.0, numba==0.48 and numpy==1.17.1 against Python 3.6, and every one of those fails on a current image. The fix is always the same four edits: numpy 1.24 removed np.float, so the face detection path needs float or np.float32; librosa 0.10 made filters.mel keyword-only, so audio.py needs mel(sr=..., n_fft=..., n_mels=...); librosa dropped .output.write_wav, so use soundfile; and torch 2.6 defaults torch.load to weights_only=True, so pass weights_only=False when loading the .pth files. If you would rather not touch any of it, use Easy-Wav2Lip (pinned to Python 3.10.11 and CUDA 12.2) or pip install lipsync. NexGPU's 2,000+ prebuilt images include a PyTorch entry that arrives with CUDA and ffmpeg already in place, which removes half the work.

Is Wav2Lip still worth using, or should I go straight to LatentSync?

It depends what you are optimising for. If you need sharpness and a commercial licence, LatentSync 1.6 is the better answer: Apache-2.0, 512x512, 18GB of inference VRAM per the official figures. The price is that 18GB floor and noticeably longer diffusion inference. Wav2Lip's seat was never image quality — it is cost and robustness. 96x96 means a negligible VRAM footprint, very fast per-frame throughput, and real tolerance for profile angles and poor source video. A common practical workflow is to run Wav2Lip first to confirm the timing is right, then re-render the locked cut through LatentSync or MuseTalk. That two-stage approach only pays off with per-second billing: on NexGPU you can prototype on an RTX 3090 at $0.193/GPU-hour and finish on an A100 PCIE 80GB at $0.824/GPU-hour without signing anything.

Can Wav2Lip run on CPU? Why rent a GPU at all?

It can — both mowshon/lipsync and wav2lip-onnx-HQ support device='cpu'. The problem is that the slow part is not the 96x96 generator, it is S3FD face detection, which has to run at full resolution on every single frame. A one-minute 25fps video is 1500 detections, and once you add upscaling you are into overnight territory on CPU. The arithmetic for renting is simple: an RTX 3090 24GB is $0.193 an hour, less than a coffee, and it turns that overnight run into minutes. NexGPU has 1,175 verified rentable nodes, 2,498 GPUs across 75 models in 51 countries and regions, reachable over SSH, Jupyter, web terminal, REST API or CLI, with bilingual support on Telegram and no ticket queue.

More in Digital humans and face animation

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.