GPU Memory Configuration for Multimodal LLMs: How Many Cards for 11B and 90B

Conclusion First: How to Position Three GPU Memory Tiers for Multimodal Models

For GPU memory configuration of multimodal LLMs, remember three key points: 11B-level vision models require at least 32GB GPUs (like RTX 5090, L40S); for constrained or high-concurrency scenarios, go with 80GB (like A100 80GB); 90B-level models require 8 cards with TP=8 tensor parallelism.

After deploying vision models to the cloud, memory planning is more error-prone than for text-only models due to three additional overhead costs. The quick reference table below can be used directly:

Below, we break down the costs into three increments: fixed, variable, and concurrency, to help you calculate accurately.

Why the Pure-Text "Parameters × 2" Formula Doesn't Work for Multimodal Memory

Many developers directly apply the rough estimation formula for text-only models. You can refer to Qwen deployment memory requirements for the text baseline. However, this formula is inaccurate for multimodal models because it misses three parts: the fixed weights of the image encoder and projection layer, the sequence expansion after a single image, and the extra KV from Cross-Attention.

For example, running an 11B vision model on a 24GB GPU (like RTX 4090): weights take up about 22GB, leaving only 2GB for the KV cache of vision tokens, leading to frequent OOM. How to calculate multimodal model memory? Breaking it down into fixed, variable, and concurrency costs is more reliable than using a formula.

First Increment (Fixed): Calculating Weights for Vision Encoder and Projection Layer

The vision encoder and projection layer are resident weights, independent of requests, occupying memory at startup. With an 11B Vision model FP16 weight of about 22GB as a baseline, this is an additional fixed cost compared to text-only models of similar parameter size (vision tower and projection layer), and this cost does not decrease with concurrency; it must be fully reserved.

There is no trick here; you must account for this static baseline based on the weight size.

Second Increment (Variable): How Many Tokens Does an Image Expand Into?

Vision models use dynamic resolution ViT architectures (like Qwen2.5-VL), encoding images into a variable number of vision tokens. The official typical range is 256~1280 tokens, with an extreme of 16384. Therefore, there is no fixed answer for how much memory image tokens consume; it depends on resolution and tiling strategy.

Higher resolution, multiple images, and long document pages multiply the sequence length, causing activations and self-attention KV to grow nonlinearly. For example, a high-resolution image may expand to over a thousand tokens, while multiple images add up directly, sharply increasing memory pressure.

Third Increment (Concurrency): Why Cross-Attention's Extra KV Explodes First

Models like Llama 3.2 Vision use cross-attention to integrate visual features, incurring extra KV overhead. With image tokens added, KV cache typically needs to be reserved at 20%~30% of the weight size. Under high concurrency, the KV pool often runs out first, not the weights.

How to estimate maximum concurrency? Reserve a KV range based on the official 20%~30% weight ratio, then divide by the KV per request (which depends on vision token count and sequence length) to get the concurrency upper bound; this must be calibrated through actual measurement, not fixed formulas.

Which Tier for 11B Models: Usable Headroom and Request Limits on 32GB GPUs

With about 22GB for weights, a 32GB GPU leaves less than 10GB for vision activations, KV cache, and framework overhead. Actual usable memory is lower after runtime overhead, so testing is required.

Therefore, 80GB GPUs are necessary for high-resolution, multi-image, and high-concurrency scenarios. As for 24GB consumer GPUs, running 11B vision models for long-context, full-resolution services is generally impractical due to too little headroom after 22GB of weights.

Why 90B Models Require Multiple GPUs: TP=8 Sharding and Memory Distribution

90B-level vision models deployed in the cloud need an 8-card cluster with TP=8 tensor parallelism. After distributing weights across 8 cards, the per-card pressure drops, but each card still bears its share of KV and activations. The exact per-card usage depends on the sharding strategy and framework implementation, requiring empirical confirmation.

Practical steps: first confirm single-node 8-card interconnect (like NVLink), then confirm the framework's TP support for vision models (vLLM multi-GPU tensor parallel configuration guide). How many GPUs for a 90B multimodal model? The answer is 8, as this is the fundamental requirement for parallel sharding.

Reverse Engineering: Four Steps to Infer GPU Tier from Business Image Specifications

Instead of copying a generic list, derive your own GPU memory configuration for multimodal LLMs using business data. Follow these four steps to infer from your business:

1. Collect actual image resolutions and the number of images per request.
2. Estimate the vision token range (using the typical 256~1280, extreme 16384).
3. Add fixed weights (e.g., 22GB) to get the static baseline.
4. Reserve KV proportion (20%~30%) for target concurrency, and map to the 32GB / 80GB / multi-card tiers.

Token count is a range, not a fixed value; it needs empirical calibration. This ensures your configuration fits your business, not guesswork.

Conduct Memory Peak Testing: What to Record for Single, Multiple, and High-Resolution Images

Recommended test records: static weight usage, peak memory after first request, KV pool level, concurrency threshold for OOM, and request latency changes.

Run three request groups on a 32GB GPU and decide whether to upgrade to 80GB or multi-card based on measured peaks. NexGPU's on-demand GPU resources allow you to rent 32GB GPUs for validation, with prebuilt models and app templates to reduce setup time, ideal for the initial phase.

Configuration Checklist and Four Common Misjudgments

After checking each item, your GPU memory configuration for multimodal LLMs will be grounded in real business needs.

Additionally, refer to GPU Out of Memory solutions for OOM issues, and Impact of FP8 and INT4 Quantization on GPU Memory to understand how quantization reduces memory usage.

FAQs

How much GPU memory does Llama 3.2 Vision 11B need?

FP16 weights are about 22GB. Adding image tokens and 20%~30% KV cache, we recommend at least 32GB. For high resolution or multi-image concurrency, 80GB GPUs are recommended. 24GB GPUs might handle simple scenarios but will OOM under long context or high concurrency.

Can 24GB GPUs run multimodal LLMs?

Yes, for small or low-resolution scenarios, but for 11B-level vision models, weights alone are 22GB, leaving insufficient space for long sequences or high concurrency. In practice, OOM is common due to insufficient KV cache. We recommend at least 32GB; for complex business needs, go with 80GB.

What GPU is recommended for Qwen2.5-VL deployment?

Qwen2.5-VL uses dynamic resolution, leading to a wide token expansion range. For normal use, 32GB GPUs are sufficient; if handling high resolution or multiple images frequently, 80GB is safer. Refer to Qwen deployment memory requirements for the text baseline.

How much memory do vision image tokens consume?

It depends on resolution and tiling strategy. Typical range is 256~1280 tokens, extreme up to 16384. Each token affects activation and KV size, causing nonlinear memory growth. We recommend estimating a range and reserving 20%~30% KV margin.

How many GPUs does a 90B multimodal model need?

It requires an 8-card cluster with TP=8 tensor parallelism. Weights are sharded across cards, and each card still bears part of the KV. A single card cannot hold the weights, so multiple cards are necessary. First confirm node interconnect and framework support.

What to do if memory is insufficient for multimodal inference?

First, reverse-engineer the GPU tier using the four steps, then test three request groups to see peak usage. If still insufficient, try reducing resolution, limiting image count, adjusting KV reserve ratio, or using quantization (like FP8) to shrink weights. For high concurrency, upgrade to 80GB or multi-card solutions.