Question 1

How much VRAM does text-generation-webui actually need?

Accepted Answer

The interface itself uses almost none. What consumes VRAM is the weights plus a preallocated KV cache. The GGUF rule of thumb holds up well: Q4_K_M ≈ params × 0.6GB, Q8_0 ≈ × 1.06, bf16 = × 2. So 8B Q4_K_M ≈ 5GB, 32B ≈ 20GB, 70B ≈ 43GB. Don't forget the cache — the docs state plainly that it is preallocated, so whatever --ctx-size you ask for is deducted immediately, and switching --cache-type from fp16 to q8_0 halves it outright. Do that math, then pick the card: NexGPU offers 75 GPU models from 24GB to 141GB — RTX 3090 at $0.193, RTX 5090 32GB at $0.723, A100 80GB at $0.824, H200 141GB at $6.660 — all billed per second, so guessing wrong just means stopping and switching.

Question 2

When should I use text-generation-webui instead of vLLM or Ollama?

Accepted Answer

The division of labour is clear. vLLM is built for high-concurrency production serving — PagedAttention, continuous batching, one backend taken to its limit. Ollama wins on simplicity. Text Generation WebUI's edge is having five backends coexist with no restart between them, plus Notebook-style continuation, conversation branching, every sampler parameter exposed, and LoRA training in the same window. It is a single-operator workbench, ideal for the model-selection and parameter-tuning phase. NexGPU's 2,000+ prebuilt images include both PyTorch and vLLM, so you can settle on a model and quant level in textgen, then move to a vLLM instance for load testing.

Question 3

What do I do when loading fails with 'unknown model architecture'?

Accepted Answer

This is one of the most common categories in the issue tracker — users have reported things like 'unknown model architecture: minimax-m3'. The cause is that each release pins llama.cpp to a fixed commit; v4.9 sits at e947228, so an architecture merged upstream afterwards won't work until the next textgen release. Three ways out: wait for the release, swap the llama.cpp binary yourself, or switch to the Transformers loader, which tracks the transformers library and usually gains new architectures first (re-estimate VRAM for bf16 or 4-bit if you do). That last route jumps you a tier in memory — spinning up an A100 PCIE 80GB at $0.824/GPU-hour on NexGPU for one evening is far faster than rebuilding your environment.

Question 4

Can I train LoRAs in it, and what card do I need?

Accepted Answer

Yes — the Training tab handles both multi-turn chat datasets and raw text, running through the Transformers path. One caveat: portable builds launch with --portable, which explicitly disables training, so you need the full start_linux.sh install (conda, Python 3.13). On memory, training has to hold weights, gradients and optimiser state at once, typically three to four times the inference footprint, so a 7B-class LoRA belongs on an 80GB card. NexGPU's H100 SXM 80GB is $3.582/GPU-hour and A100 SXM4 80GB is $1.088/GPU-hour — stop when the run finishes, with no quota request for a few hours of fine-tuning.

Question 5

Why can't I find the ExLlamaV2 or AutoGPTQ loaders my tutorial mentions?

Accepted Answer

Because they are no longer in the loader list. The v4.9 Model tab offers exactly five: llama.cpp, ik_llama.cpp, Transformers, ExLlamav3_HF / ExLlamav3, and TensorRT-LLM. The wave of 2024-era GPTQ tutorials is out of date, and EXL3 is ExLlamaV3's new format — it is not interchangeable with EXL2 weights. If your library is all older quantisations, converting to GGUF and running llama.cpp is the safest migration. This is exactly the kind of trial-and-error that belongs on per-second billing: a NexGPU Tesla T4 16GB is $0.298/GPU-hour, enough to work through format compatibility one file at a time.

Question 6

How do I safely reach the web UI and API when it's deployed in the cloud?

Accepted Answer

v4.9 already tightened CORS to localhost by default and fixed a path-traversal issue in character-name handling, so the right posture is simply not to expose it publicly. An SSH port forward for 7860 and 5000 gives you zero configuration and zero attack surface. For team access, pair --listen with --gradio-auth user:password and set --api-key on the API side — don't skip either layer. NexGPU instances ship with SSH, Jupyter, a web terminal and a REST API by default, and with 51 countries and regions to choose from you can put the node close to your users. If something won't connect, reach us on Telegram in English or Chinese — no ticket queue.

Version	Parameters	VRAM	Context	Notes
llama.cpp (GGUF)	1B–70B, the single-card workhorse	Q4_K_M ≈ params × 0.6GB: 8B ≈ 5GB / 32B ≈ 20GB / 70B ≈ 43GB; Q8_0 ≈ × 1.06	--ctx-size defaults to 0, reading the native window from GGUF metadata	v4.9 ships commit e947228. --gpu-layers defaults to -1 (auto) and spills to system RAM when the model doesn't fit; --cache-type accepts fp16, q8_0 and q4_0. The cache is preallocated, so whatever context you request is deducted up front.
ik_llama.cpp (separate ik- package)	Large MoE models and CPU hybrid offload	Same order as the equivalent GGUF, but --cpu-moe / --n-cpu-moe move expert layers to system RAM and can cut VRAM to well under half the weight size	Same as GGUF	v4.9 ships commit 40254a5, packaged separately as textgen-portable-ik-4.9-* assets. For 100B+ sparse MoE models it is far more VRAM-frugal than mainline llama.cpp, at the cost of being bottlenecked by memory bandwidth.
ExLlamaV3 / ExLlamav3_HF (EXL3)	8B–70B, pure-GPU inference	Straight bpw math: 32B @ 4.0bpw ≈ 16GB, 70B @ 3.0bpw ≈ 26GB, plus KV	Cache supports fp16 and q2–q8, with separate K and V bit widths	v4.9 ships ExLlamaV3 0.0.34. Pure GPU — there is no CPU offload fallback, so if it doesn't fit, loading simply fails. The non-_HF ExLlamav3 variant additionally supports speculative decoding and multimodal models.
Transformers (bf16 / bitsandbytes)	1B–13B at full precision, or 4-bit / 8-bit	bf16 = params × 2GB (8B ≈ 16GB); load_in_4bit brings 8B down to ≈ 6GB, a bit lower with use_double_quant	Bounded by both --ctx-size and the model's native window	The only path that supports LoRA training, and the one with the broadest architecture coverage. When llama.cpp hasn't merged support for a new architecture yet, Transformers is often the only thing that will load it at all.
TensorRT-LLM	Precompiled engines	Determined by the precision and batch/seq limits chosen at compile time, then fixed	Baked in when the engine is built	NVIDIA-only and the highest single-card throughput available here, but changing model or context length means recompiling the engine. Suited to a settled, long-running service — not to experimentation.

Self-hosting Text Generation WebUI: five backends, one window — and the VRAM math behind it

Five loaders and their quant formats: who eats how much VRAM

Pick the card by what you actually intend to run

Deploying Text Generation WebUI on NexGPU

Start a GPU instance

Install text-generation-webui

Drop the model into user_data/models/ and launch with the API on

Reach it over an SSH tunnel, not --share

What a month actually costs, with the arithmetic

Frequently asked questions