Multimodal vision-language model

Self-hosting InternVL3.5: from a 5GB Q4 file to an eight-GPU flagship, with the VRAM math shown

OpenGVLab's open multimodal line spans nine sizes from 1B to 241B-A28B. What each tier actually needs, which card to rent, and the gotchas that stop your first run — all checkable numbers.

InternVL · self-hosted

InternVL is maintained by OpenGVLab (Shanghai AI Laboratory) at github.com/OpenGVLab/InternVL under the MIT license, with weights published on Hugging Face under Apache-2.0 — no email-for-access, no commercial-use form. That is the first reason it keeps winning private-deployment bake-offs. The current line is InternVL3.5, released 26 August 2025, in nine sizes: dense 1B, 2B, 4B, 8B, 14B and 38B; MoE 30B-A3B and 241B-A28B; plus a GPT-OSS-based 20B-A4B preview. Every size also ships Pretrained, Instruct, MPO and final CascadeRL checkpoints, and a parallel set of -HF format repos. Check the suffix before you start the download.

Architecturally it is a straight ViT–MLP–LLM stack: InternViT-300M for 1B through 14B and 30B-A3B, InternViT-6B for 38B and 241B-A28B (counted as 5.5B vision parameters), with the language tower initialized from the Qwen3 series. Images go through dynamic high-resolution tiling: each 448px tile encodes to 1024 visual tokens, then pixel shuffle compresses it to 256. The reference code uses max_num=12, so a single image can become 12 tiles plus one thumbnail — 13 × 256 = 3,328 visual tokens worst case. Nine out of ten VRAM blowups come from that number, not from the weights. Weights are fixed; tile count is a knob.

Three real changes separate InternVL3.5 from InternVL3. Cascade RL — offline MPO followed by online RL — is credited with a +16.0% reasoning gain. ViR, the Visual Resolution Router, scores each patch for semantic richness and routes it to either 256 or 64 tokens; packaged as the Flash branch it halves visual tokens while retaining close to 100% of full performance. DvD, decoupled vision-language deployment, splits the vision tower and the LLM across separate GPUs for a stated 4.05× speedup over InternVL3. One more thing worth stating plainly: the line still stops at 3.5 — there is no InternVL4, and InternVL3.5-Flash, added at the end of September 2025, is an efficiency branch of the same generation, not a successor. If you are still searching for InternVL2.5 or InternVL3 deployment guides, jump straight to 3.5; the serving stack and call pattern are largely compatible at the same size.

01 —

InternVL3.5 VRAM by size

bf16 figures come from official weights and measured community GGUF file sizes; quantized rows name the exact quant type

Version	Parameters	VRAM	Context	Notes
InternVL3.5-1B / 2B / 4B	0.8B / 2B / 5B (includes 0.3B InternViT)	bf16 roughly 1.6GB / 4GB / 10GB by parameter count; community GPTQ INT4 exists for 1B and 2B	32K (SFT stage)	Edge boxes and bulk pre-filtering. Use these to discard the 90% of images that never needed a big model, then escalate the rest.
InternVL3.5-8B	8.5B (0.3B InternViT-300M + 8.2B Qwen3)	bf16 16.4GB / Q8_0 8.71GB / Q5_K_M 5.85GB / Q4_K_M 5.03GB	32K (SFT stage)	The sweet spot. Largest dense size you can ship on one GPU, and the default starting point for OCR, invoice extraction and chart QA.
InternVL3.5-30B-A3B	31B total / ~3B activated (MoE)	bf16 61.1GB / Q8_0 32.5GB / Q4_K_M 18.63GB	32K (SFT stage)	30B of knowledge at 3B of activated compute. Q4_K_M at 18.63GB makes it the best value tier on a 32GB card.
InternVL3.5-38B	38B (5.5B InternViT-6B + 32.8B Qwen3)	bf16 65.5GB / Q8_0 34.8GB / Q4_K_M 19.8GB / Q3_K_M 16.0GB	32K (SFT stage)	Swaps in the 6B vision tower — that is where the gains on fine-grained charts and dense page layouts come from. Official guidance: two A100s.
InternVL3.5-241B-A28B	240.7B total / 28B activated (5.5B ViT + 235.1B Qwen3)	~480GB of bf16 weights; the official deployment baseline is 8 × A100 80GB	32K (SFT stage)	The flagship, described by its authors as state of the art among open-source MLLMs and closing the gap to frontier commercial models.
InternVL3.5-Flash, all sizes (1B–241B-A28B)	Same as the matching mainline size	Same weight footprint, but half the visual tokens — prefill VRAM and latency drop with it	32K (SFT stage)	The ViR-routed build: a stated 50% cut in visual tokens while keeping close to 100% of InternVL3.5 performance. Pick it for long documents and high-resolution screenshots.

02 —

Pick a card by workload — real NexGPU rates

Matched against InternVL's actual weight footprint and visual-token cost. No 38B on a 24GB card here.

8B at Q4_K_M, single GPU, validating OCR and document extraction quality
RTX 3090 24GB$0.193/GPU-hour
5.03GB of weights plus thirteen tiles of visual tokens leaves plenty of headroom in 24GB, and Ampere does bf16 natively — fewer dtype headaches than the similarly priced P40 or V100.
8B in full bf16 in production, multi-image inputs at 32K context
RTX 5090 32GB$0.723/GPU-hour
Leaves a dozen-plus GB for KV cache and visual tokens on top of 16.4GB of weights. A 24GB card holds the weights but OOMs the moment you batch multiple images.
30B-A3B or 38B in bf16 on a single GPU
A100 PCIE 80GB$0.824/GPU-hour
Fits 30B-A3B's 61.1GB and 38B's 65.5GB without sharding. The MoE tier only activates 3B, so throughput lands near an 8B while the knowledge is 30B-class.
241B-A28B flagship node, or high-concurrency multi-GPU 38B serving
H100 SXM 80GB × 8$3.582/GPU-hour ($28.656/hour for the node)
8 × 80GB is the official baseline. NexGPU nodes go up to 14 GPUs and 2,152GB of node VRAM, so you assemble this in one box instead of talking across machines.

03 —

Four steps to a running InternVL3.5

From a bare instance to an OpenAI-compatible endpoint. Only one parameter actually requires thought.

01
Start an instance and pull the weights
Pick a NexGPU machine with a prebuilt vLLM or PyTorch image (both are in the 2,000+ image catalogue), then pull weights over SSH or Jupyter. 8B is about 17GB, 30B-A3B about 61GB — size the disk accordingly. Note that -HF repos are native transformers format while the unsuffixed ones use InternVL's own format; they take different serving paths, so do not mix them.
```
huggingface-cli download OpenGVLab/InternVL3_5-8B --local-dir ./InternVL3_5-8B
```
02
Serve it with LMDeploy
LMDeploy is InternVL's sibling engine, and both the TurboMind and PyTorch backends cover InternVL3.5 from 1B to 241B-A28B. Use --tp 2 for 38B and --tp 8 for 241B-A28B. The one exception is the GPT-OSS-20B-A4B preview: LMDeploy has no GPT-OSS support, so that tier must go through vLLM.
```
lmdeploy serve api_server OpenGVLab/InternVL3_5-8B --backend turbomind --server-port 23333 --tp 1
```
03
Or take the vLLM / SGLang route
vLLM covers the line through the InternVLChatModel architecture and needs trust_remote_code. For multi-image work you must set limit-mm-per-prompt explicitly — the default allows a single image, and that is where most first multi-image calls fail. SGLang exposes an OpenAI-compatible endpoint as well.
```
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code --max-model-len 32768 --limit-mm-per-prompt '{"image": 8}'
```
04
Tune the tile count and pin your VRAM
This is the most valuable line in the whole deployment. The reference code uses max_num=12, meaning one image can become 12 tiles of 448px plus a thumbnail: 13 × 256 = 3,328 visual tokens. Dropping to max_num=6 halves both the visual tokens and their KV footprint, with almost no loss on invoices, forms and screenshots; only newspaper-dense layouts and engineering drawings genuinely need 12. Also keep the standard trio: trust_remote_code=True, torch_dtype=torch.bfloat16, use_flash_attn=True.
```
pixel_values = load_image('invoice.png', max_num=6).to(torch.bfloat16).cuda()
```

An actual invoice

Say you run InternVL3.5-8B in bf16 over 200,000 receipts. Take an RTX 5090 32GB at $0.723/GPU-hour, which leaves a dozen-plus GB for multi-image KV on top of the 16.4GB of weights. Twenty hours of wall clock: 20 × $0.723 = $14.46. Weights plus intermediate output occupy 60GB of storage; at the $0.414/GB-month median, twenty hours is 20/720 = 0.028 of a month, so 60 × $0.414 × 0.028 ≈ $0.69. Export 5GB of results at the $0.0081/GB median egress rate = $0.04. Total: $15.19 for 200,000 pages. Run the same batch at Q4_K_M on an RTX 3090 24GB instead and it is $0.193 × 20 = $3.86 — trade accuracy against cost as you see fit. The flagship prices separately: 241B-A28B on 8 × A100 SXM4 80GB is 8 × $1.088 = $8.704/hour, so a three-hour evaluation run costs $26.11; the same node on 8 × H100 SXM is 8 × $3.582 = $28.656/hour, faster but more than three times the price and rarely worth it for evaluation. Billing is metered per second and priced per hour, with no minimum, no setup fee and no quota request. Compute billing stops when the instance stops; storage keeps accruing until you destroy it.

04 —

FAQ

How much VRAM does InternVL3.5-8B actually need? Is a 24GB card enough?

The bf16 weights measure 16.4GB, so on weights alone a 24GB card fits. But InternVL's VRAM is dominated by visual tokens: at max_num=12 a single image is 3,328 tokens, and batching a few images drains 24GB fast. Practical answer: Q4_K_M at 5.03GB is comfortable on 24GB, while bf16 in production wants 32GB or more. On NexGPU that is $0.193/GPU-hour for an RTX 3090 24GB and $0.723/GPU-hour for an RTX 5090 32GB, billed per second — rent an hour and test both paths before you commit.

Are there GGUF quants of InternVL? Can I run it in llama.cpp or Ollama?

Yes, and coverage is thorough. bartowski, lmstudio-community and mradermacher all publish InternVL3.5 GGUFs from 1B up to 241B-A28B. Measured sizes: 8B Q4_K_M is 5.03GB and Q8_0 is 8.71GB; 30B-A3B Q4_K_M is 18.63GB; 38B Q4_K_M is 19.8GB and Q3_K_M is 16.0GB. You need the matching mmproj file for the vision tower — pull only the main weights and you get a text-only model. To compare three or four quant levels in one sitting, a NexGPU RTX 3090 24GB at $0.193/GPU-hour is more than enough.

Why does AWQ / W4A16 quantization fail for InternVL3.5 under LMDeploy?

It is not your config. In LMDeploy's support matrix, InternVL v1.5 through InternVL3 are marked as W4A16-capable, but the InternVL3.5 row reads No — for both the TurboMind and PyTorch engines. Two workable 4-bit paths remain: community GPTQ W4A16 weights served through vLLM, or GGUF through llama.cpp. Experiments like this cost close to nothing on per-second billing; NexGPU has no minimum rental and no quota request, so you can kill an instance ten minutes in.

How does InternVL3.5-Flash differ from the regular build, and is it worth switching?

Flash inserts the ViR Visual Resolution Router between the vision tower and the LLM. It scores each patch for semantic richness and routes it to 256 or 64 tokens, halving visual tokens while the authors report retaining close to 100% of performance. Weight size is unchanged; what you save is prefill compute and KV memory. The rule of thumb is simple: switch to Flash for high-resolution documents, web screenshots and multi-page PDFs; skip it for single small images. Flash exists for every size from 1B to 241B-A28B, so spin up two identical NexGPU instances and A/B them in an hour.

How many GPUs does InternVL3.5-38B really need? Will a single 80GB card do?

The official model card recommends two A100s. Look at the numbers, though: bf16 weights measure 65.5GB, leaving a dozen-plus GB on a single 80GB card. For offline evaluation at short context and small batch, one A100 PCIE 80GB at $0.824/GPU-hour genuinely works. The moment you serve concurrent traffic or feed multi-image long context, KV eats that headroom and you should use --tp 2. NexGPU nodes carry up to 14 GPUs, so both 80GB cards sit in the same box with no cross-machine hop.

Is there anything newer than InternVL3.5? I'm on InternVL2.5 — should I upgrade?

As things stand the line still ends at InternVL3.5, released 26 August 2025. There is no InternVL4, and InternVL3.5-Flash, added at the end of September 2025, is an efficiency branch of the same generation. The InternVL3 (1B–78B) and InternVL2.5-MPO (1B–78B) repos are still up, so existing services can stay where they are. The reasons to move are the +16.0% reasoning gain from Cascade RL and the 4.05× deployment speedup from DvD, both of which are noticeable on document and chart workloads. To bench old against new before committing, an A100 PCIE 80GB on NexGPU is $0.824/GPU-hour billed per second — running both sides costs next to nothing.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.