Skip to main content
NexGPU

Avatar · audio-driven talking head video

Self-hosting Hallo: one photo, one audio track, and four generations of VRAM math

Fudan's Generative Vision Lab has shipped four generations of Hallo, swapping the backbone from SD 1.5 all the way to a video DiT. The VRAM floor went from 9.77GB to 80GB. Pick the wrong one and a 60-second clip costs you ten times more than it should.

Sign up freeView pricing

Hallo does exactly one thing: hand it a front-facing portrait and a speech track, and it returns video where the lips, expression and head pose all follow the audio. The original paper — Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation — landed in June 2024 on a Stable Diffusion 1.5 denoising UNet, with AnimateDiff's mm_sd_v15_v2 motion module, a wav2vec2-base-960h audio encoder, and the paper's actual contribution: HADVS, which routes audio features into three separate cross-attention paths for lips, expression and pose. That is why the inference script exposes --lip_weight, --face_weight and --pose_weight as independent dials — you can tighten the mouth without amplifying the head sway.

The next three releases each pushed in a different direction. Hallo2 (ICLR 2025) kept the SD 1.5 backbone and fixed the two things that made v1 a demo rather than a tool: patch-drop augmentation extends continuous generation to as long as an hour, and a video super-resolution head derived from CodeFormer takes the output to 4K. Hallo3 (CVPR 2025) replaced the backbone outright, swapping the UNet for CogVideoX-5B-I2V's video diffusion transformer — background dynamics and large head motion improve dramatically, at the cost of a 52GB weight set that OOMs on a 48GB card. Hallo4 (SIGGRAPH Asia 2025, from Fudan with Baidu, Nanjing University and Alibaba) moved again, this time onto Wan2.1-1.3B, using Direct Preference Optimization to push fidelity with a backbone four times smaller than Hallo3's.

So "how much VRAM does Hallo need" is not one number — it is four numbers that span nearly an order of magnitude. This page lays out the weight sizes, measured VRAM, clip lengths, and the things that actually trip people up when self-hosting: the English-only wav2vec2 encoder and what it does to Chinese lip sync, unnormalised audio producing inconsistent results, Hallo3 having no multi-GPU inference path at all, and Hallo4 loading an 11GB umT5-XXL text encoder for a task that is driven entirely by audio. Each row maps to a specific per-second-billed NexGPU machine.

01 —

Four generations: weight size, measured VRAM, clip length

Taken from the official repos, the HuggingFace file manifests and the paper tables — not estimated

VersionParametersVRAMContextNotes
Hallo (v1, arXiv June 2024, MIT)SD 1.5 UNet backbone + HADVS | net.pth 4.85GB, ~11.3GB total weightsfp16 256×256 ≈ 6.62GB | 512×512 ≈ 9.77GB | 1024×1024 ≈ 20.66GB512×512 · 25fps · 16-frame window + 2 motion frames · 40 steps · CFG 3.5The cheapest tier to run — any 24GB card is comfortable. The VRAM figures come from Table 7 of the paper, alongside 0.46 / 1.63 / 10.29 second timings. Use it to validate your pipeline and source material before spending real money.
Hallo2 (ICLR 2025, MIT + S-Lab 1.0)Same SD 1.5 backbone net.pth 4.85GB + SR head net_g.pth 905MB | ~13GB totalGeneration stage on par with v1 at ~10GB; the 4K upscale is a separate second pass through CodeFormer + RealESRGAN_x2plus, frame by frameUp to 1 hour of continuous generation · 512×512 base · up to 4K after upscalingThe generation you would actually ship long-form content with. Two stages: inference_long.py produces the 512×512 cut, then video_sr.py upscales it separately. Budget the SR pass on its own — it walks every frame again.
Hallo3 (CVPR 2025, MIT, bound by the CogVideoX licence)CogVideoX-5B-I2V DiT + 3D VAE + T5-v1.1-XXL | main checkpoint 29.1GB, ~52GB totalbf16 inference: repo issues report OOM at 48GB, OOM on an A100 even with xformers, and an RTX 4090 24GB stalling at 51-step sampling — plan for a single 80GB cardsampling_num_frames 13 (4× temporal compression in the 3D VAE, i.e. 49-frame clips) · 25fps · 1:1 or 3:2 reference imagesThe best-looking and most expensive generation. There is no multi-GPU inference path — issue #13 shows 4× RTX 6000 Ada (48GB each) using only one card, saturated by 15 seconds of audio. Even training has OOM'd on an A800 80GB.
Hallo4 (SIGGRAPH Asia 2025, under the Wan2.1 licence)Wan2.1-1.3B backbone model_weight.ckpt 5.54GB + umT5-XXL encoder 11.36GB + Wan2.1_VAE 508MB | ~17.8GB totalbf16 from ~32GB (small backbone, but the 11.36GB text encoder is resident too); caching the T5 embeddings brings it onto an RTX 5090 32GBReference images from 1:1 to 480:832 portrait · Wan 480p tierThe current line. DPO aligns the model directly against human preference for fidelity, on a backbone over four times smaller than Hallo3's — which puts VRAM back into consumer-card territory. Best value of the four.

02 —

Which GPU per generation, with NexGPU per-hour rates

Sized against measured peaks — we will not pretend a 5B video DiT fits on a 24GB card

  • Hallo / Hallo2 at 512×512, validating footage and batch generation

    RTX 3090 24GB$0.193/GPU-hr

    The measured peak is 9.77GB; 24GB holds the UNet, the motion module and the onnxruntime face sessions with half the card to spare — and it is the cheapest GPU on the list.

  • Hallo at 1024×1024, or the Hallo2 long-video plus 4K upscale pipeline

    RTX 5090 32GB$0.723/GPU-hr

    1024×1024 measures at 20.66GB and 10.29 seconds, which leaves a 24GB card uncomfortably tight. 32GB gives you the buffer, and the SR pass is brute-force per-frame work where newer silicon converts directly into fewer billed hours.

  • Hallo4 (Wan2.1-1.3B + DPO) inference

    RTX A6000 48GB$0.817/GPU-hr

    The 5.54GB backbone, the 11.36GB umT5-XXL and the VAE all sit in VRAM at once; 48GB runs it unpatched. Once you have cached the T5 embeddings, drop to a 5090 32GB and save.

  • Hallo3 (CogVideoX-5B DiT) inference and long-clip stitching

    A100 PCIE 80GB$0.824/GPU-hr

    48GB is confirmed insufficient and Hallo3 has no multi-GPU inference path, so adding cards will not save you — a single 80GB card is the floor. Single-card inference does not need NVLink, so PCIE saves $0.264/hr over SXM4.

03 —

Boot to finished clip in four steps

Following the Hallo2 long-video pipeline, with the other generations' differences noted inline

  1. 01

    Spin up an instance and build the environment

    NexGPU's prebuilt PyTorch images ship CUDA and conda, so you go straight to the env. All four generations want Python 3.10, but the CUDA target is not uniform — Hallo, Hallo3 and Hallo4 document CUDA 12.1 while Hallo2 documents 11.8, and mixing torch builds across them fails most often at xformers. ffmpeg is a hard dependency for frame extraction and muxing; do not skip it.

    git clone https://github.com/fudan-generative-vision/hallo2 && cd hallo2 && conda create -n hallo python=3.10 -y && conda activate hallo && pip install -r requirements.txt && apt-get install -y ffmpeg
  2. 02

    Pull the weights onto the data volume, with the exact directory layout

    The Hallo2 set is about 13GB: hallo2/net.pth at 4.85GB, the SR head net_g.pth at 905MB, the SD 1.5 UNet at 3.44GB, the AnimateDiff motion module mm_sd_v15_v2.ckpt at 1.82GB, sd-vae-ft-mse at 335MB, wav2vec2-base-960h at 378MB, plus the InsightFace scrfd / glintr100 / 1k3d68 ONNX set and Kim_Vocal_2.onnx for vocal separation. Hallo3 is 52GB; Hallo4 is roughly 17.8GB. The scripts resolve models by hard-coded paths, so the tree under pretrained_models must match the README exactly — this is the single most common first-run failure.

    huggingface-cli download fudan-generative-ai/hallo2 --local-dir ./pretrained_models
  3. 03

    Generate the 512×512 cut first

    long.yaml defaults to 512×512, fp16, 40 steps, CFG 3.5, 25fps, 16-frame clips with 2 motion frames. Three hard rules on source images: square crop, face occupying 50–70% of the frame, front-facing with under 30° rotation — use --face_expand_ratio if yours does not fit. Audio must be WAV; background music is fine because Kim_Vocal_2.onnx separates the vocal track before wav2vec2 sees it — but normalise the volume first, since the repo lists audio level affecting inference results as an open item. v1 uses scripts/inference.py, Hallo3 uses bash scripts/inference_long_batch.sh or python hallo3/app.py for the Gradio UI, and Hallo4 runs bash inf.sh.

    python scripts/inference_long.py --config ./configs/inference/long.yaml
  4. 04

    Run the 4K upscale separately, then stop the instance

    The upscale is a full second pass over every frame, so budget it on its own and only run it once you are happy with the 512×512 cut. -w is CodeFormer's fidelity weight (1 favours fidelity, 0 favours quality), -s 4 is the scale factor, --face_upsample enhances the face region separately. Copy the output off, then stop the instance — compute billing halts immediately. Storage keeps accruing at $0.414/GB-month until you destroy the volume. Egress is $0.0081/GB, so a one-hour 4K render at roughly 18GB costs about $0.15 to pull down.

    python scripts/video_sr.py --input_path ./output_long/debug/result.mp4 --output_path ./output_sr --bg_upsampler realesrgan --face_upsample -w 1 -s 4

What a 60-second talking-head clip actually costs

Take the 512×512 default used by Hallo and Hallo2. Table 7 of the paper reports 9.77GB and 1.63 seconds on A100-class hardware (issue #28 in the repo questions whether that timing is per-frame; the community reads it as per-frame, and one run on your own instance settles it). At 1.63 s/frame and 25fps: a 60-second clip is 1,500 frames, so 1,500 × 1.63 s = 2,445 s ≈ 0.68 hours. On an A100 PCIE 80GB at $0.824/GPU-hr that is 0.68 × $0.824 ≈ $0.56 of compute, plus roughly 10 minutes to pull weights and warm up at 0.17 × $0.824 ≈ $0.14 — about $0.70 all in. Now run the same job on an RTX 3090 24GB at $0.193/GPU-hr: 9.77GB fits with room to spare, wall-clock stretches — call it 2.5× longer, so ~1.7 hours — and 1.7 × $0.193 ≈ $0.33. Slower, and less than half the price per finished clip. That is exactly why we default this tier to the 3090. Hallo3 is a different ledger entirely: 48GB is not enough, so you start at an A100 PCIE 80GB burning $0.824/GPU-hr from second one, and the 52GB weight set alone costs 52 × $0.414 ÷ 30 ≈ $0.72/day just sitting on disk. Billing is metered per second with no minimum, no setup fee and no quota request, and compute stops the moment the instance does — so the cheapest sequence is to get the pipeline and the source material right on a 3090, then switch to the 80GB card only for the final render.

04 —

FAQ

How much VRAM does Hallo need to self-host? Is a 24GB card enough?

It depends entirely on the generation. For v1 and Hallo2 at the 512×512 default, the paper measures a 9.77GB peak, so 24GB is generous; 256×256 needs only 6.62GB, and 1024×1024 needs 20.66GB, which fits on 24GB but leaves very little headroom. Hallo3 is a different animal: repo issues report OOM at 48GB, OOM on an A100 even with xformers enabled, and an RTX 4090 24GB stalling during 51-step sampling — plan on a single 80GB card. Hallo4 starts around 32GB because the 11.36GB umT5-XXL stays resident. On NexGPU an RTX 3090 24GB is $0.193/GPU-hr and an A100 PCIE 80GB is $0.824/GPU-hr, billed per second — booting one for twenty minutes to watch nvidia-smi is faster than asking a forum.

Which one should I use — Hallo, Hallo2, Hallo3 or Hallo4?

Just validating the effect on a few seconds of audio: v1, lightest on VRAM. Producing minutes-to-an-hour of long-form talking head and delivering in 4K: Hallo2, which exists for precisely those two problems. Chasing maximum background dynamics and large head motion with budget as a secondary concern: Hallo3. Wanting the best fidelity-per-gigabyte without being chained to a 52GB weight set and an 80GB card: Hallo4, on the Wan2.1-1.3B backbone with DPO alignment. If you are undecided, boot an RTX 3090 on NexGPU at $0.193/GPU-hr, get Hallo2 working end to end, and decide from there whether Hallo3 is worth A100 money.

Does Hallo work with Chinese or other non-English audio?

This is the most common surprise. All four repos specify English WAV input, and the v1 roadmap still lists "Mandarin Chinese support" as TBD. The reason is the audio encoder: wav2vec2-base-960h was trained on 960 hours of English LibriSpeech, so non-English phonemes land outside its distribution. Feed it Chinese and you will get video, but the lip shapes go soft — plosives and retroflex sounds worst of all — and there is no official non-English checkpoint. The upside is that training scripts and datasets are open (Hallo3 released 70+ hours of talking-head video), so fine-tuning on your own language is a real path. That is the ideal workload for per-second billing: spin up an A100 80GB or an H100 SXM 80GB on NexGPU, run the fine-tune, stop the instance, and skip buying a card for a one-off job.

Can I run Hallo3 across several 24GB or 48GB cards?

No, and this is the thing to know before you commit. Issue #13 documents 4× RTX 6000 Ada at 48GB each where inference uses only one card, saturated by 15 seconds of audio — there is no official multi-GPU inference path, so adding cards does not solve VRAM. A single 80GB card is the only route. On NexGPU the A100 PCIE 80GB at $0.824/GPU-hr is the value pick, since single-card inference has no use for NVLink and there is no reason to pay $1.088 for SXM4. If you are running the two-stage SFT fine-tune instead, move up to the H100 SXM 80GB at $3.582/GPU-hr — we go up to 14 GPUs per node with a 2,152GB node VRAM ceiling.

How long does the Hallo2 4K upscale take, and is it worth it?

Understand first that it is a genuinely separate second pass: inference_long.py produces the 512×512 cut, and only then does video_sr.py walk every frame through CodeFormer plus RealESRGAN_x2plus. You pay for that compute all over again. So the right order is to review the 512×512 output, confirm the lip sync and expression are right, and only then upscale — never burn hours upscaling a take you will throw away. On flags: -w 1 favours fidelity and lower values favour perceived quality, -s 4 sets the scale, --face_upsample enhances the face separately. This two-stage shape is made for per-second billing: on NexGPU, run pass one on an RTX 3090 at $0.193/GPU-hr and pass two on an RTX 5090 32GB at $0.723/GPU-hr, matching each stage to the right card instead of renting one expensive GPU for both.

What is the licence — can I use Hallo commercially?

Check each one separately. The v1 and Hallo2 repos are MIT, but Hallo2's super-resolution module is adapted from CodeFormer and that portion carries the S-Lab License 1.0. The Hallo3 repo is MIT, yet it is a fine-tuned derivative of CogVideoX-5B and must satisfy CogVideoX's original terms too. Hallo4 is a derivative of Wan2.1-1.3B and is governed by the WAN LICENSE. All four also carry the same ethics notice: audio-driven portrait animation carries deepfake risk, and you should obtain informed consent from anyone whose likeness you animate. Take the compliance question to your own counsel — our job is to remove every excuse on the compute side. NexGPU spans 51 countries and regions, 1,175 verified rentable nodes, 2,498 GPUs and 75 GPU models, reachable over SSH, Jupyter, web terminal, REST API and CLI, with 2,000+ prebuilt images including PyTorch and ComfyUI, and bilingual support on Telegram with no ticket queue.

More in Digital humans and face animation

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.