Skip to main content
NexGPU

Audio-Driven Avatars

Self-host AniPortrait: turn a wav file into a talking portrait on a single 24GB GPU

One reference image plus one audio track gives you a 512x512 talking video with matched lip sync and natural head motion. The full weight bundle is 10.3GB, inference runs in fp16, and one card does the whole pipeline — as long as that card isn't a 10GB one.

Sign up freeView pricing

AniPortrait comes out of Tencent Games Zhiji (Huawei Wei, Zejun Yang, Zhisheng Wang). The paper is arXiv:2403.17694, "AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation", and the code lives at github.com/Zejun-Yang/AniPortrait under Apache-2.0. It works in two stages: wav2vec2-base-960h encodes the audio, audio2mesh reconstructs a 3D facial mesh and audio2pose generates head pose, and both get projected into a 2D landmark sequence. That sequence then conditions a Stable Diffusion v1.5-derived diffusion stack — a reference UNet that captures the source portrait's appearance, a denoising UNet carrying an AnimateDiff-style motion module for temporal coherence, and a pose guider that injects the landmarks. The repo openly credits Moore-AnimateAnyone, AnimateDiff, magic-animate and EMO for the lineage.

The VRAM arithmetic follows directly from that structure. In the official weight repo ZJYang/AniPortrait you get denoising_unet.pth and reference_unet.pth at 3.44GB each, motion_module.pth at 1.82GB, pose_guider.pth at 670MB, audio2pose.pt at 482MB, audio2mesh.pt at 382MB and film_net_fp16.pt at 69MB — about 10.3GB, before you add the SD 1.5 base model, sd-vae-ft-mse, the 1.22GB image_encoder and wav2vec2-base-960h. The config sets weight_dtype to fp16, so resident weights land in the 7GB range, roughly half the on-disk footprint. Issue #130 in the repo records a 10GB card hitting CUDA OOM outright (10.00 GiB total, 8.97 GiB already allocated at the crash). Worse, the requirement grows with clip length: pipeline_pose2vid_long infers over a 16-frame sliding window with 4 frames of overlap, but the full latent and the full pose condition tensor stay resident, and the VAE decodes frame by frame and concatenates on-device. Sixty seconds at 30fps is 1,800 frames — those two buffers alone add roughly 8GB.

One more thing worth saying plainly: the repo's last commit landed on 2 July 2024, and the changelog stops at the 21 April 2024 audio2pose weight release. The field has moved since — Sonic (CVPR 2025, tested on a single 32G GPU, but CC BY-NC-SA 4.0 and therefore non-commercial), EchoMimicV3 (Ant Group, 1.3B params, 12G VRAM quantised at 768x768), Tencent's own HunyuanVideo-Avatar (24GB minimum for 704x768x129f, 10GB once TeaCache landed), and Hallo3 from Fudan and Baidu (a CogVideoX-5B I2V fine-tune, tested on H100). People still deploy AniPortrait for two very concrete reasons: Apache-2.0 on the code, and a pinned SD1.5 stack that reproduces end to end on one 24GB card. All you need is the right GPU — which is what NexGPU is for: 1,175 verified nodes, 2,498 GPUs, 75 models, metered per second, stopped the moment you stop.

01 —

Four inference paths, two training stages, and what each one costs in VRAM

AniPortrait doesn't ship version numbers — what you actually pick is a run mode and a weight set

VersionParametersVRAMContextNotes
scripts.audio2vid (audio-driven)SD1.5 UNet 0.86B x2 + motion module + wav2vec2-base-960h~7GB resident in fp16; 16GB floor and 24GB comfortable for a 512x512 ten-second clip; 10GB OOMs in practice16-frame sliding window, 4-frame overlap; motion module temporal position encoding capped at 32The main path: one reference image plus a wav gives you a talking video. Defaults are --steps 25, --cfg 3.5, --seed 42, -W 512 -H 512.
scripts.vid2vid (face reenactment)Same stack plus MediaPipe 468-point face mesh extractionSame class as audio2vid; scales linearly with the source video's total frame countOutput follows the source video's native fps unless --fps overrides itThe 2 April 2024 pose retargeting update: source pose is normalised against the reference image's initial translation, then smoothed over a 3-frame window, so it tracks even when reference and driving head positions differ a lot.
scripts.pose2vid (self-driven, landmark sequence)pose_guider 670MB + denoising UNetSame class as audio2vid, minus roughly 0.9GB by skipping audio2mesh and audio2pose16-frame sliding window, context_batch_size defaults to 1Drive directly from an existing landmark sequence and skip the whole audio chain — the cheapest route for batch re-renders and A/B comparisons.
audio2pose.pt (head pose model)482MB standalone weightAlready counted in the ~7GB resident figureGenerates per-frame head Euler angles and translationOnly released on 21 April 2024, and off by default — you must delete the pose_temp entry from animation_audio.yaml to enable it. Leave it in and every clip reuses the fixed head_pose_temp/pose_temp.npy template, so head motion looks identical every time.
-acc with film_net_fp16.pt (FILM interpolation)69MB frame interpolation networkUnder 1GB on topfi_step defaults to 3 — diffusion runs on a third of the frames, the rest are interpolatedAdded 7 April 2024. Roughly a 3x speedup, at the cost of smearing fast lip detail. Turn it off for final renders.
train_stage_1 / train_stage_2 (fine-tuning on your own data)Stage 1 trains reference UNet, denoising UNet and pose guider; stage 2 trains the motion module only80GB class. Stage 1 uses train_bs 2 at 512^2; stage 2 uses train_bs 1 x 16 frames with gradient checkpointing and 8-bit Adam already onsample_n_frames 16, sample_size [512, 512]max_train_steps is 300000 and 40000 respectively, and stage 2 additionally needs mm_sd_v15_v2.ckpt. Issue #208 reports OOM even across multiple 4090s — put this one on 80GB cards.

02 —

Which GPU to rent — pick by clip length and job, not by how new the card is

AniPortrait pins torch 2.0.1 and CUDA 11.7, so newer silicon is not automatically better here

  • Getting the pipeline running and iterating on ten-second clips

    RTX 3090 24GB$0.193/GPU-hour

    Ampere sm_86 is natively covered by torch 2.0.1, so requirements.txt installs as written, and 24GB leaves real headroom for the 16-frame window plus the full-clip condition tensor.

  • Rendering a minute or more of audio in one pass, without chunking

    RTX A6000 48GB$0.817/GPU-hour

    VRAM grows linearly with frame count — the pose condition tensor and concatenated VAE output for 1,800 frames add about 8GB on their own. 48GB means you never have to rewrite the pipeline to fit.

  • High-volume batch rendering where unit cost is what matters

    Tesla V100 32GB$0.188/GPU-hour

    sm_70 is fully supported under CUDA 11.7, and the pipeline is fp16 throughout so nothing here wants bf16. You get 8GB more than a 3090 at the lowest rate on the fleet.

  • Stage-1 / stage-2 fine-tuning on your own character data

    A100 SXM4 80GB$1.088/GPU-hour

    Stage 2 pushes a 16-frame temporal window through every step and still blows past multiple 4090s with gradient checkpointing and 8-bit Adam enabled. 80GB is the honest starting point.

03 —

From cold boot to first video, in four steps

Start from a PyTorch prebuilt image and route around the two stale instructions in the README

  1. 01

    Boot a 3090, clone the repo, install the pinned deps

    Pick a PyTorch image with Python 3.10+ and CUDA 11.7, then SSH in. requirements.txt is pinned hard: torch==2.0.1, torchvision==0.15.2, xformers==0.0.22, diffusers==0.24.0, transformers==4.30.2, numpy==1.24.4, mediapipe==0.10.11. Do not casually bump numpy to 2.x — mediapipe and decord both break immediately.

    git clone https://github.com/Zejun-Yang/AniPortrait && cd AniPortrait && pip install -r requirements.txt
  2. 02

    Pull the 10.3GB weight bundle plus the base models

    The project weights come from ZJYang/AniPortrait in one shot. The base model is where the README has gone stale: runwayml/stable-diffusion-v1-5 has been taken down from HuggingFace, so use the community mirror stable-diffusion-v1-5/stable-diffusion-v1-5 instead. You also need sd-vae-ft-mse, the image_encoder folder (1.22GB) from lambdalabs/sd-image-variations-diffusers, and facebook/wav2vec2-base-960h, all arranged into ./pretrained_model/ exactly as the README's directory tree shows — the scripts use hardcoded relative paths.

    huggingface-cli download ZJYang/AniPortrait --local-dir ./pretrained_model && huggingface-cli download stable-diffusion-v1-5/stable-diffusion-v1-5 --local-dir ./pretrained_model/stable-diffusion-v1-5
  3. 03

    Run the audio-driven path and get a ten-second clip out

    Point the test_cases block in configs/prompts/animation_audio.yaml at your reference image and wav. Delete the pose_temp entry to switch on audio2pose and get real head motion; leave it and you get the fixed template every time. -L sets the frame count — 300 frames is exactly ten seconds at 30fps — and -acc enables FILM interpolation for roughly 3x faster renders. Drop -acc for final output.

    python -m scripts.audio2vid --config ./configs/prompts/animation_audio.yaml -W 512 -H 512 -L 300 -acc
  4. 04

    Launch the Gradio UI, or switch to face reenactment

    scripts.app brings up the official Gradio interface, which is what you hand to teammates who don't want to touch the CLI. For face reenactment, use vid2vid instead — it extracts 3D landmarks from the driving video with MediaPipe and retargets them onto your reference face. When you're done, stop the instance: compute billing stops immediately, and the weights stay on storage so the next boot picks up where you left off.

    python -m scripts.app

What one ten-second avatar clip actually costs

Take the RTX 3090 24GB at $0.193/GPU-hour. Step one — install, pull the 10.3GB project weights and roughly 5GB of base models — runs about 0.5 hours, so 0.5 x $0.193 = $0.097. Then iteration: assume a 10-second clip (300 frames, of which -acc leaves roughly 100 to actually diffuse, at 25 steps and cfg 3.5) takes 6 minutes on a 3090 — measure this yourself once you're on the box rather than trusting the assumption — and 30 test clips is 3 hours, so 3 x $0.193 = $0.579. Together that's $0.676: sixty-eight cents for thirty usable clips, about $0.023 each. At production volume, 10 clips per hour puts unit cost at $0.0193, so 1,000 clips is roughly $19.3. For long-form audio move to the RTX A6000 48GB at $0.817/GPU-hour; a one-minute render estimated at 35 minutes is 0.583 x $0.817 = about $0.476 per clip, and you skip the chunk-and-stitch rework entirely. Two tails to remember: weights plus dependencies occupy about 25GB, and storage runs $0.414/GB-month, so 25 x 0.414 = $10.35/month — compute billing stops when the instance stops, but storage keeps accruing until you destroy it. Egress is $0.0081/GB, and 1,000 512x512 MP4s is around 2GB, so about $0.016 — noise. End to end, this is two orders of magnitude cheaper than buying a card for the job.

04 —

FAQ

How much VRAM does AniPortrait really need? Will a 12GB card work?

12GB is the danger zone. The hard data point is issue #130 in the repo: a 10GB card hits CUDA OOM outright, with 10.00 GiB total capacity and 8.97 GiB already allocated at the crash. Working from weight sizes, fp16 residency is around 7GB; the rest goes to activations for the 16-frame context window, the full-clip pose condition tensor (roughly 1.5MB per 512x512 fp16 frame) and the frame-by-frame VAE decode output concatenated on device (roughly 3MB per frame in fp32) — and both of those scale linearly with clip length. The pipeline exposes enable_sequential_cpu_offload() but doesn't use it by default, and forcing it makes runs slow enough to be pointless. So: 16GB is the floor and 24GB the comfortable choice for 512x512 ten-second clips; go straight to 48GB past a minute. On NexGPU that's $0.193/GPU-hour for an RTX 3090 24GB and $0.817/GPU-hour for an RTX A6000 48GB, billed per second — cheaper to find out than to buy hardware and guess.

Is AniPortrait still maintained? In 2026 should I use it or move to Sonic or EchoMimicV3?

Honestly: the last commit is 2 July 2024 and the changelog stops at the audio2pose weights on 21 April 2024, so treat it as frozen. The current field includes Sonic (CVPR 2025, tested on a single 32G GPU), EchoMimicV3 (Ant Group, 1.3B, 12G quantised at 768x768), HunyuanVideo-Avatar (24GB minimum for 704x768x129f, 10GB with TeaCache) and Hallo3 (CogVideoX-5B I2V fine-tune, tested on H100). But AniPortrait holds two positions nobody has taken from it: Apache-2.0 code you can ship commercially — Sonic is CC BY-NC-SA 4.0 and explicitly non-commercial — and a fully pinned dependency set that reproduces end to end on one 24GB card. The sensible move is to bench all of them side by side before committing. Every target GPU for that comparison, from 24GB to 141GB, is on NexGPU under one console, so you don't change vendors halfway through.

I followed the README exactly and it still won't run. What are the usual failures?

Four, in rough order of frequency. First, the base model the README points at, runwayml/stable-diffusion-v1-5, has been removed from HuggingFace — use the mirror at stable-diffusion-v1-5/stable-diffusion-v1-5. Second, numpy is pinned to 1.24.4, and anything that quietly upgrades it to 2.x breaks mediapipe 0.10.11 and decord 0.6.0. Third, the default torch 2.0.1 wheel is a CUDA 11.7 build, so your GPU architecture has to fall inside what that toolkit supports. Fourth, the weight directory tree has to match the README exactly, because the scripts use hardcoded relative paths. Booting from a NexGPU PyTorch prebuilt image removes the driver and CUDA layer from the equation; work through the other three and it usually comes up first try.

It crashes with None as soon as I swap in a different reference image. Is that a format problem?

Not format — it's a single-face constraint. LMKExtractor in src/utils/mp_utils.py uses MediaPipe's face_landmarker_v2_with_blendshapes.task to pull 468 landmarks and a set of blendshape coefficients, and it is configured for one face only: if detection fails, or if a second face appears anywhere in the frame, it returns None and the rest of the pipeline follows it down. A portrait poster in the background, a group photo, a second face in a mirror will all trigger it. Crop to a clean single-subject headshot before feeding it in. Iterations like this take seconds, and NexGPU meters per second — spin up a 3090, get your inputs clean, shut it down for less than the price of a coffee.

I have an RTX 5090 32GB — more VRAM than a 3090. Why isn't it your top recommendation for AniPortrait?

Because this codebase is pinned to torch==2.0.1 on CUDA 11.7, and the Blackwell architecture (sm_120) needs a CUDA 12.8-class toolchain and a far newer PyTorch to have executable kernels at all — run it as-is and you will most likely get "no kernel image is available for execution on the device". You can upgrade torch, but diffusers 0.24.0, transformers 4.30.2 and xformers 0.0.22 are pinned to the same era and will cascade, which is a separate project's worth of work. If you want this to just run, pick an architecture the stack was built for: NexGPU has the RTX 3090 24GB at $0.193/GPU-hour, the Tesla V100 32GB at $0.188/GPU-hour and the RTX A6000 48GB at $0.817/GPU-hour, all comfortably inside that dependency envelope.

Can I use AniPortrait output commercially? What should I check on licensing?

The code repo is Apache-2.0, which is more permissive than much of this field. But you have to stack the base models on top: it depends on Stable Diffusion v1.5, which is CreativeML OpenRAIL-M and carries use restrictions, while wav2vec2-base-960h and the image_encoder have their own licence pages — walk through each one with legal before you ship. By comparison, Sonic is CC BY-NC-SA 4.0 and explicitly bars commercial use, which is often the deciding factor that sends teams back to AniPortrait. Likeness and voice rights are a separate matter entirely; clearing your reference images and audio is on you. What we can do is make the experimentation cheap: NexGPU spans 51 countries and regions across 1,175 verified nodes, with SSH, Jupyter, REST API and CLI access, and bilingual support over Telegram with no ticket queue.

More in Digital humans and face animation

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.