3D generation model

Self-hosting LGM: 10GB of VRAM for image-to-3D, and the model itself is not what eats it

LGM is 0.415B parameters and 830MB in fp16 — three models fit on one 24GB card with room to spare. The hard parts are two CUDA extensions you compile on the spot, one checkpoint filename the readme gets wrong, and an nvdiffrast backend that always fails on a headless box.

LGM · self-hosted

LGM stands for Large Multi-View Gaussian Model, from Jiaxiang Tang (ashawkey) and colleagues at Peking University and NTU S-Lab, paper arXiv:2402.05054, code at 3DTopia/LGM, weights at ashawkey/LGM, both MIT. The approach is refreshingly direct: no SDS optimisation, no direct mesh regression. A multi-view diffusion model turns one image (or one prompt) into four views at 0°/90°/180°/270°, then an asymmetric U-Net emits 14 channels per pixel — 3 for position, 1 for opacity, 3 for scale, 4 for the rotation quaternion, 3 for colour. That is a complete 3D Gaussian. Four 128×128 feature maps stitch into 65,536 Gaussians; the paper claims a 3D object in under 5 seconds at a training render resolution of 512.

So it matters where the VRAM actually goes. LGM's own fp16 weights are just 830MB. The ~10GB figure in the official readme is the whole pipeline: ImageDream for image-to-3D (guidance 5.0) or MVDream SD2.1 for text-to-3D (guidance 7.5) running 30 diffusion steps, plus the U2-Net inside rembg, plus LPIPS VGG when training, plus the 360° orbit video render at the end. Pull LGM out on its own for an inference service and the footprint drops sharply — but run the official scripts as shipped and you should budget 10GB. A 24GB card is the comfortable landing spot, not because the pipeline won't fit in less, but because you still need headroom for the mesh-fitting pass in convert.py.

One more thing worth saying plainly: LGM is February 2024 work, and the News section of its readme stops at the 2024.4.3 rotation-normalisation fix. The front rank of open 3D generation has since moved to TRELLIS (image-large 1.2B, text-xlarge 2.0B, at least 16GB VRAM per the docs, MIT) and Tencent's Hunyuan3D 2.1 (DiT 3.0B plus Paint 1.3B, 6GB for shape and 16GB for shape with texture). LGM is not obsolete, though — it occupies a different niche. 0.4B parameters, an 830MB checkpoint, MIT with no extra clauses, single-card seconds-per-asset Gaussians, and a codebase short enough to read end to end. For bulk previews, asset triage, the front end of a Gaussian splatting pipeline, or as a starting point for teaching and modification, it is still the best value on the board.

01 —

Checkpoints and configs: don't grab the wrong file

Only the big config has public weights, and only one of the four safetensors files is the one you want.

Version	Parameters	VRAM	Context	Notes
big + model_fp16_fixrot.safetensors	0.415B (415,042,848)	830MB weights / ~10GB full pipeline	splat 128 → 4×128×128 = 65,536 Gaussians, render 512	The only version to use. Fine-tuned for 30 more epochs after the severe rotation-normalisation bug was fixed on 2024.4.3. Always spell this filename out in --resume.
big + model_fp16.safetensors (superseded)	0.415B	830MB weights	same, splat 128 / render 512	The pre-fix weights with the rotation bug. The trap: the readme's example commands still name this file, so copy-pasting silently degrades your output with no error at all.
model_fixrot.safetensors (fp32)	0.415B	1.66GB weights, resident as F32	splat 128 / render 512	Start here if you want to keep training LGM, distil it, or change the architecture. Pointless for pure inference — infer.py calls model.half() anyway.
lrm / small config (no public weights)	same architecture, one fewer decoder stage	training in bf16, batch 8/GPU	splat 64 → 16,384 Gaussians, render 256, 12-view supervision	The paper's default settings, up_channels (1024,1024,512,256). No weights were released, so this one you train yourself.
tiny config (no public weights)	down channels cut to (32,64,128,256,512), count not published	training batch 16/GPU	splat 64 → 16,384 Gaussians, render 256, 8 views	The ablation architecture. Good for proving your training pipeline works end to end, or for data experiments on cheap cards.

02 —

Which card to rent

Inference starts at 10GB, but mesh conversion and training reproduction are entirely different weight classes.

Run image-to-3D and text-to-3D, output .ply plus a 360° video
RTX 3090 24GB$0.193/GPU-hour
A 10GB pipeline sits easily in 24GB, and the Tesla T4 16GB costs $0.298 — more money for less memory, so there is no case for it.
Batch asset generation plus convert.py Gaussian-to-textured-mesh
RTX 4090 24GB$0.540/GPU-hour
Mesh conversion is a 512-step fit_nerf plus a 2048-step fit_mesh optimisation loop — pure compute, and Ada throughput cuts minutes per asset.
Keep a Gradio service resident alongside TRELLIS or Hunyuan3D 2.1 for comparison
RTX A6000 48GB$0.817/GPU-hour
48GB holds the full LGM stack and a 16GB-class newer model at the same time, so you skip the cold start of swapping weights.
Reproduce training with the official acc_configs/gpu8.yaml
A100 SXM4 80GB$1.088/GPU-hour
The big config is bf16, batch 8/GPU, 8-view supervision plus LPIPS VGG — memory and interconnect both matter. NexGPU nodes go up to 14 GPUs, so 8 come up at once.

03 —

Four steps to a running install

torch 2.1.0 + cu118 is the officially verified combination, and two CUDA extensions must be compiled on the box.

01
Boot a 24GB card and pin torch and xformers first
The readme says it outright: xformers is required. Versions must line up with torch and CUDA, so install torch 2.1.0 + cu118 before xformers — reverse the order and pip drags torch back to the default build. NexGPU's prebuilt PyTorch images boot ready, and per-second billing means this compile phase costs cents.
```
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118 && pip install -U xformers --index-url https://download.pytorch.org/whl/cu118
```
02
Compile diff-gaussian-rasterization and nvdiffrast
LGM does not use the stock Gaussian rasteriser — it needs ashawkey's fork with depth and alpha rendering, and it must be cloned with --recursive or the submodule is missing and the build fails outright. nvdiffrast only matters for mesh export, but install it now and be done. Both want nvcc, so an image with the CUDA toolkit already present saves real time.
```
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization && pip install ./diff-gaussian-rasterization && pip install git+https://github.com/NVlabs/nvdiffrast && pip install -r requirements.txt
```
03
Pull only the rotation-fixed checkpoint
830MB, about a minute. ImageDream (ashawkey/imagedream-ipmv-diffusers) and MVDream SD2.1 (ashawkey/mvdream-sd2.1-diffusers) download automatically on first run — note they load with trust_remote_code=True, so an air-gapped environment needs the cache warmed in advance.
```
mkdir -p pretrained && wget -P pretrained https://huggingface.co/ashawkey/LGM/resolve/main/model_fp16_fixrot.safetensors
```
04
Infer the Gaussians, then bake a mesh with a 1024 texture
infer.py runs rembg matting, recenter at border_ratio 0.2, 30 ImageDream diffusion steps and one LGM forward pass, writing a .ply and an .mp4. convert.py is the slow part: Gaussians are fit to a NeRF (512 steps), marching cubes extracts a surface, fit_mesh runs 2048 steps down to a 50k-face target, and fit_mesh_uv bakes a 1024×1024 texture. On a headless cloud box --force_cuda_rast is not optional, it is mandatory.
```
python infer.py big --resume pretrained/model_fp16_fixrot.safetensors --workspace workspace_test --test_path data_test && python convert.py big --test_path workspace_test/saved.ply --force_cuda_rast
```

What one full run actually costs

Price it out on an RTX 3090 24GB at $0.193/GPU-hour. First boot compiles diff-gaussian-rasterization and nvdiffrast and pulls the ImageDream and MVDream weights — call it 0.5 hours: 0.5 × $0.193 = $0.0965. Then three hours of continuous batch image-to-3D: 3 × $0.193 = $0.579. Then an hour of convert.py turning the Gaussians you kept into textured glb: 1 × $0.193 = $0.193. Total 4.5 hours × $0.193 = $0.8685 — under a dollar to stand up an entire 3D generation pipeline from nothing and come away with usable assets. The same 4.5 hours on an RTX 4090 24GB ($0.540/GPU-hour) is $2.43, and the extra two dollars buy back time on the diffusion sampling and the mesh fitting. Storage bills separately: LGM's fp16 weights are 830MB, and with ImageDream, MVDream and your .ply/.mp4/.glb output, 20GB works out to 20 × $0.414 = $8.28/month. Compute billing stops the moment the instance stops, but storage keeps accruing until you destroy it — so export the workspace and delete it. For a genuine training reproduction, the official launcher is acc_configs/gpu8.yaml: A100 SXM4 80GB at $1.088/GPU-hour means 8 × $1.088 = $8.704/hour for the node. NexGPU goes up to 14 GPUs per node, with no quota request and no minimum.

04 —

FAQ

How much VRAM does LGM really need? Is 8GB enough?

The readme says roughly 10GB, and that covers ImageDream, MVDream and LGM all resident together — 8GB fails at the multi-view diffusion stage, not at LGM. LGM's fp16 weights are only 830MB, so you can absolutely split the diffusion front end onto another card or process and run LGM alone on something small. For almost everyone, though, booting one RTX 3090 24GB at $0.193/GPU-hour and running the whole stack beats two hours of refactoring: on NexGPU those two hours of machine time cost $0.386.

Why does convert.py die on nvdiffrast / OpenGL on my server?

nvdiffrast defaults to an OpenGL backend that wants a real display context, and cloud GPU instances are headless — no X server, no EGL device. The fix is already in LGM's core/options.py: pass --force_cuda_rast to switch to the CUDA rasteriser. This is the single most common trap when moving LGM to the cloud. NexGPU images ship the full CUDA toolkit, so with that flag it runs straight through.

What is the difference between model_fp16.safetensors and model_fp16_fixrot.safetensors?

fixrot is the version re-finetuned for 30 epochs after the severe rotation-normalisation bug was fixed on 3 April 2024; the other is the pre-fix checkpoint. The trap is that the readme's example command still reads --resume pretrained/model_fp16.safetensors while the download instruction fetches the fixrot file, so copy-pasting silently loads the wrong weights and yields mushy Gaussians with no error. Always write model_fp16_fixrot.safetensors in full. Spin up a per-second-billed instance and A/B it — the difference costs pennies to see for yourself.

Does LGM output Gaussians or a mesh? Can I export glb for Blender or Unity?

infer.py produces a 3D Gaussian .ply and a 360° orbit mp4 — not a mesh. For glb you run convert.py, which is a chain of Gaussians → NeRF → marching cubes → mesh fitting → UV baking: 512 fit_nerf steps, 2048 fit_mesh steps with periodic remeshing and decimation to 50k faces, then 512 fit_mesh_uv steps baking a 1024×1024 texture. That is a minutes-long optimisation loop, not a single forward pass, which is exactly why it dominates compute in batch production. It is also why we point that stage at an RTX 4090 24GB at $0.540/GPU-hour.

Is LGM still worth using, and how does it compare to TRELLIS or Hunyuan3D?

It depends on what you want. For the best geometry and PBR materials, go to TRELLIS (image-large 1.2B, at least 16GB VRAM per the docs, MIT) or Hunyuan3D 2.1 (DiT 3.0B plus Paint 1.3B, 6GB for shape, 16GB with texture). For the fastest, lightest, most hackable option, LGM still has no replacement: 0.4B parameters, an 830MB checkpoint, MIT with no extra clauses, and code short enough to read in a sitting — ideal for bulk previews and asset triage. The practical engineering answer is to run both tracks: boot one RTX A6000 48GB on NexGPU at $0.817/GPU-hour, hold LGM and a 16GB-class newer model at once, and bench them head to head on your own inputs instead of trusting someone else's demo reel.

How many GPUs do I need to train LGM myself, and what about the dataset?

The official launcher is acc_configs/gpu8.yaml, so eight GPUs. The big config is bf16, batch 8/GPU, lr 4e-4, 30 epochs, 8-view supervision with an LPIPS VGG loss at 512 render resolution — the memory pressure is real. Watch the data: the original dataloader reads from AWS S3 and cannot be used as-is elsewhere, so you must rewrite core/provider_objaverse.py. The roughly 80K Objaverse subset used for training is public at ashawkey/objaverse_filter. On hardware, A100 SXM4 80GB is $1.088/GPU-hour, so an 8-GPU node is $8.704/hour. NexGPU allows up to 14 GPUs per node with a maximum node VRAM of 2,152GB, billed per second with no quota request and no setup fee, so trying a single epoch is a bounded expense.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.