Vision-Language Model

Self-hosting Qwen-VL: the VRAM bill from 2B to 235B

The short answer: what you actually want to deploy today is Qwen3-VL. The 2B is 1.11GB at Q4_K_M; the 235B-A22B is 142GB at the same quant. That hundred-fold spread is the whole GPU decision.

Qwen-VL · self-hosted

"Qwen-VL" means two different things. Narrowly, it is the August 2023 release in QwenLM/Qwen-VL: a Qwen-7B language tower bolted to a ViT, fixed 448×448 image input, shipped as Qwen-VL, Qwen-VL-Chat and Qwen-VL-Chat-Int4. Running at 448 rather than 224 was unusual at the time and gave it real fine-grained OCR and bounding-box grounding, but that repo now points visitors to later generations. Broadly, Qwen-VL is a whole lineage: Qwen-VL (2023) → Qwen2-VL (2024, naive dynamic resolution plus M-RoPE) → Qwen2.5-VL (January 2025, 3B/7B/32B/72B, Apache 2.0) → Qwen3-VL (September–October 2025). If you are standing up a private deployment now, the answer is Qwen3-VL, not the 2023 weights.

The more useful thing to know is that the -VL suffix has been retired. Qwen3-VL is the last generation to carry it: four dense sizes (2B, 4B, 8B, 32B) plus two MoE sizes (30B-A3B, 235B-A22B), each in Instruct and Thinking editions, 256K native context extensible to 1M with YaRN, built on Interleaved-MRoPE, DeepStack and Text–Timestamp Alignment, all under Apache 2.0. By Qwen3.5 in February 2026, vision moved into the trunk: 0.8B, 2B, 4B, 9B, 27B, 35B-A3B, 122B-A10B and 397B-A17B are every one of them natively multimodal via early fusion over a Gated DeltaNet plus sparse-MoE hybrid, with no separate VL repo at all. Qwen3.6-27B / 35B-A3B and Qwen3.8-27B followed the same pattern. So "the latest Qwen-VL" depends on what you want: the mature ecosystem (Qwen3-VL) or the newest capability (Qwen3.8-27B).

What actually breaks self-hosted deployments is never the weight file — it is visual tokens. Dynamic resolution means a single 4K screenshot expands into thousands of tokens that all land in the KV cache. The guidance Qwen2.5-VL published is a budget between min_pixels = 256×28×28 and max_pixels = 1280×28×28; leave it uncapped and a model whose weights occupy 5GB will still OOM on the third image. Budget VRAM as weights plus visual-token allowance plus KV cache, in that order. NexGPU rents the entire ladder — RTX 3090 24GB at $0.193/GPU-hr through H200 141GB at $6.660/GPU-hr, up to 14 GPUs per node and 2,152GB max node VRAM — metered per second and priced per hour, so tuning max_pixels for two hours costs less than a coffee.

01 —

Every Qwen3-VL size, with real footprints

Quantised sizes are the published file sizes from the official GGUF repos. They are weights only — leave headroom for the mmproj vision tower, the KV cache and your visual-token budget.

Version	Parameters	VRAM	Context	Notes
Qwen3-VL-2B / 4B-Instruct (Thinking editions too)	2B / 4B	2B: Q4_K_M 1.11GB / Q8_0 1.83GB / F16 3.45GB. 4B: Q4_K_M 2.5GB / Q8_0 4.28GB / F16 8.05GB	256K native, 1M with YaRN	The sweet spot for edge boxes and high-volume OCR pipelines. One 24GB card holds several replicas for concurrent document parsing.
Qwen3-VL-8B-Instruct / Thinking	8B (Hugging Face counts the weights as 9B)	Q4_K_M 5.03GB / Q8_0 8.71GB / F16 16.4GB	256K native, 1M with YaRN	The workhorse for single-GPU private deployments. F16 at 16.4GB sits right at the edge of a 24GB card and runs comfortably with context capped near 32K.
Qwen3-VL-30B-A3B-Instruct / Thinking (MoE)	30B-A3B	Q4_K_M 18.6GB / Q8_0 32.5GB / F16 61.1GB	256K native, 1M with YaRN	30B total but roughly 3B active per token. You pay VRAM on the total and get latency on the active count — pick it when throughput matters more than peak quality.
Qwen3-VL-32B-Instruct / Thinking	32B dense	Q4_K_M 19.8GB / Q8_0 34.8GB / F16 65.5GB	256K native, 1M with YaRN	The strongest dense tier for visual reasoning — complex charts, GUI agent tasks and 3D grounding pull clearly ahead of the 8B. An official FP8 build is also published.
Qwen3-VL-235B-A22B-Instruct / Thinking (plus official FP8)	235B-A22B	Q4_K_M 142GB / Q8_0 250GB / F16 470GB	256K native, 1M with YaRN	The flagship. Note that Transformers cannot load the official FP8 weights directly — you must serve them through vLLM or SGLang, and the reference example runs at gpu_memory_utilization=0.70.
Qwen3.5 / Qwen3.6 / Qwen3.8-27B (the natively multimodal successors)	0.8B–397B-A17B / 27B / 35B-A3B	Qwen3.5-9B: ~6.5GB at 4-bit, ~13GB at 8-bit, ~19GB BF16. Qwen3.5-27B: ~17GB at 4-bit, ~54GB BF16. Qwen3.8-27B: 16–19GB at 4-bit, ~31GB at 8-bit, ~56GB BF16	262,144 native, up to 1,010,000 with YaRN	No VL suffix — vision is fused into the base model. Qwen3.8-27B interleaves Gated DeltaNet with Gated Attention layers under Apache 2.0. Its sibling Qwen3.8-2.4T-A95B is text-only and ships under a proprietary licence, so do not shortlist it as a vision model.

02 —

Pick the card by the way you actually run it

Size VRAM as quantised weights plus mmproj plus visual tokens plus KV cache — never by the weight file alone.

Qwen3-VL-2B / 4B at Q4_K_M — wiring up the pipeline, OCR and invoice batches
RTX 3090 24GB$0.193/GPU-hr
The 4B at Q4_K_M is only 2.5GB, leaving 20GB-plus for visual tokens and concurrent KV — the cheapest place on the network to get a multimodal pipeline wrong a few times.
Qwen3-VL-8B at F16 in production, 32K context, moderate concurrency
RTX 4090 24GB$0.540/GPU-hr
16.4GB of weights fits, and Ada's FP8 path can halve that again for throughput. If you need 128K context or heavy multi-image batching, step up to the RTX 5090 32GB at $0.723/GPU-hr.
Qwen3-VL-32B at Q4_K_M (19.8GB) or 30B-A3B at Q8_0 (32.5GB) for a real evaluation
RTX A6000 48GB$0.817/GPU-hr
48GB lets Q8 weights and a long-context KV cache coexist, so you are not dropping to Q4 just to fit in 32GB. For the 32B at F16 (65.5GB), move to an A100 SXM4 80GB at $1.088/GPU-hr.
Qwen3-VL-235B-A22B official FP8, full private deployment
H200 141GB × 2$6.660/GPU-hr
282GB across two cards holds FP8 weights plus KV with room to spare at 2 × 6.660 = $13.32/hr — cheaper than four H100 SXM 80GB at 4 × 3.582 = $14.328/hr, and one less interconnect hop.

03 —

Getting Qwen3-VL running on NexGPU

PyTorch and vLLM are already in the 2,000+ prebuilt images — no CUDA archaeology required.

01
Start an instance and get a shell
Pick a node from the 1,175 verified rentable ones and boot a vLLM or PyTorch image. You get SSH, Jupyter, a web terminal, a REST API and a CLI. Compute billing starts the second the instance does.
```
ssh root@<node-ip> -p <port>
```
02
Pull the weights and get Transformers to the right version
Qwen3-VL requires transformers 4.57.0 or newer; at launch that had not shipped to PyPI, so the official instruction was to install from the main branch. Models from Qwen3.5 onward want main-branch vLLM and SGLang for the same reason. ModelScope mirrors are faster from Asian nodes.
```
pip install git+https://github.com/huggingface/transformers && hf download Qwen/Qwen3-VL-8B-Instruct --local-dir /workspace/qwen3-vl-8b
```
03
Serve an OpenAI-compatible endpoint
One vLLM command exposes /v1/chat/completions and accepts images as image_url parts. Cap max-model-len at 32K first, watch the memory curve, then raise it; the official FP8 example runs at gpu-memory-utilization 0.70. SGLang is the same idea via python3 -m sglang.launch_server --model-path ... --host 0.0.0.0 --port 30000.
```
vllm serve Qwen/Qwen3-VL-8B-Instruct --gpu-memory-utilization 0.70 --max-model-len 32768 --port 8000
```
04
Going GGUF to save VRAM? Do not forget mmproj
This is the single most common Qwen3-VL self-hosting mistake: llama.cpp needs two files, the language-model GGUF and the mmproj vision projector. Download only the first and you get a model that cannot see. Pass --mmproj explicitly to llama-mtmd-cli or llama-server. Sampling: Instruct wants temp 0.7 / top_p 0.8 / top_k 20; Thinking wants temp 1.0 / top_p 0.95 / top_k 20.
```
llama-server -m Qwen3VL-8B-Instruct-Q4_K_M.gguf --mmproj mmproj-Qwen3VL-8B-Instruct-F16.gguf --ctx-size 8192 -ngl 99 --port 8080
```

What a full Qwen3-VL selection run actually costs

The sane path is small first, large second. Start on an RTX 3090 24GB at $0.193/GPU-hr with Qwen3-VL-4B at Q4_K_M (2.5GB) to get prompts, image preprocessing and your max_pixels ceiling right: 2 hours = 2 × 0.193 = $0.386. Once the pipeline holds, move to an RTX A6000 48GB at $0.817/GPU-hr and run the real evaluation on Qwen3-VL-32B at Q4_K_M (19.8GB): 6 hours = 6 × 0.817 = $4.902. Keep 60GB of weights and datasets around for three days so you can re-run: 60 × 0.414 × 3 ÷ 30 = $2.48. Export 8GB of outputs and logs: 8 × 0.0081 = $0.065. Add it up — 0.386 + 4.902 + 2.48 + 0.065 = $7.83 — and under eight dollars buys you a real 4B-versus-32B comparison on your own data. If you want to benchmark the flagship too, two H200 141GB cards running the 235B-A22B FP8 come to 2 × 6.660 = $13.32/hr, so a four-hour bake-off is $53.28; the same job on four H100 SXM 80GB would be 4 × 3.582 = $14.328/hr, which is more. Compute billing stops the second the instance stops, storage keeps running until you destroy the volume, so delete what you are done with. No minimum, no setup fee, no quota request.

04 —

FAQ

Is Qwen-VL still usable, and which generation should I actually self-host?

The 2023 Qwen-VL / Qwen-VL-Chat weights are still downloadable, but fixed 448×448 input and a short context no longer match what document understanding demands. For production, take Qwen3-VL: the last generation with the VL name, six sizes from 2B to 235B-A22B, Apache 2.0, and by far the most mature tooling. If you want the newest capability instead, go straight to the natively multimodal Qwen3.8-27B. Either way you need GPUs, and NexGPU rents 75 GPU models across 2,498 cards from RTX 3090 24GB up to H200 141GB — so changing model generations never means changing providers.

How much VRAM does Qwen3-VL-8B need? Is a 24GB card enough?

The official GGUF repo publishes Q4_K_M at 5.03GB, Q8_0 at 8.71GB and F16 at 16.4GB — weights only. Add the mmproj vision tower, the KV cache and your visual tokens, and 24GB runs F16 reliably with max-model-len held near 32K, while Q8_0 leaves plenty of slack. For long context or heavy multi-image batching, 32GB is the calmer choice. On NexGPU the RTX 4090 24GB is $0.540/GPU-hr and the RTX 5090 32GB is $0.723/GPU-hr, billed by the second, so an hour is enough to settle the question empirically.

Why does it OOM on images when the weights are only 5GB?

Because Qwen's VL models use native dynamic resolution: the bigger the image, the more visual tokens it produces, and every one of them lands in the KV cache. A high-resolution screenshot easily consumes over a thousand tokens. Qwen2.5-VL's published budget runs from min_pixels = 256×28×28 to max_pixels = 1280×28×28 — set the ceiling before you talk about concurrency, and for video clamp fps alongside max_pixels. Finding the right values takes iteration, and NexGPU meters per second with no minimum, so you only pay for the forty minutes you actually spend tuning.

I'm running Qwen3-VL under llama.cpp and it can't see images. What's wrong?

Almost certainly you downloaded the language-model GGUF and skipped the mmproj vision projector. The official GGUF repos ship mmproj separately in FP16 and Q8_0, and it only loads if you pass --mmproj to llama-mtmd-cli or llama-server. Separately, the Thinking chat templates had a known llama.cpp failure that crashed on the second turn ("Value is not callable"); re-uploaded quants with the corrected template fix it, and Ollama support for new-generation Qwen multimodal GGUFs has consistently lagged. NexGPU images ship llama.cpp and vLLM ready to go, which skips most of this.

How many GPUs does a private Qwen3-VL-235B-A22B deployment need?

It depends on precision. The GGUF builds are 142GB at Q4_K_M, 250GB at Q8_0 and 470GB at F16, and the official FP8 weights cannot be loaded by Transformers at all — vLLM or SGLang only, with gpu_memory_utilization=0.70 in the reference example. In practice two H200 141GB cards (282GB) are the least painful FP8 setup at 2 × 6.660 = $13.32/hr, while Q4_K_M will come up on two A100 SXM4 80GB at $1.088/GPU-hr. NexGPU nodes go up to 14 GPUs and 2,152GB of node VRAM, so multi-GPU topologies need no special request.

Do I need to redo the VRAM maths for fine-tuning, or for moving to Qwen3.5?

Yes to both. Full fine-tuning wants roughly three to four times the bf16 weight footprint; LoRA and QLoRA add only a few GB on top of inference. Generations differ too: Qwen3.5-9B is about 6.5GB at 4-bit and 19GB at BF16, Qwen3.5-27B about 17GB at 4-bit and 54GB at BF16, and Qwen3.8-27B lands at 16–19GB at 4-bit and roughly 56GB at BF16 — and in that line the smaller models have reasoning off by default. NexGPU spans 51 countries and regions across 1,175 verified nodes, so you can fine-tune on an A100 80GB, drop back to a 4090 for inference, and get bilingual help on Telegram with no ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.