How to Choose GPU Memory? Calculate Weight and KV Cache Usage in 4 Steps

Before diving into spec sheets, the answer to how to choose GPU memory comes from working backward from your workload: account for model weights, KV cache, intermediate tensors, and framework overhead separately, then add a margin for engine version and memory fragmentation. The vLLM v0.27.0 release in August 2026 rewrote both the byte width of the KV cache and memory usage during startup, proving that memory needs aren't fixed across engine versions—which is precisely why you must leave headroom.

Before Calculating, Answer Three Questions: Model Size, Context Length, and Concurrency

Three input variables are essential for memory estimation: model parameter count, target context length, and concurrent requests. Talking about "how much memory a 7B model needs" without the latter two is bound to be wrong, because the KV cache grows almost linearly with context and concurrency. Before starting, gather these: model weight file size (to infer parameter count and precision), target context length (e.g., 8K or 32K tokens), and maximum concurrent requests during inference.

First Calculation: Weight Memory—FP16/FP8/INT4 Conversion

Weight memory = parameter count × bytes per parameter. FP16 takes 2 bytes, FP8 takes 1 byte, INT4 takes 0.5 bytes per parameter. For a 7B model, FP16 weights are ~14GB, FP8 ~7GB, INT4 ~3.5GB. Note these are weights only, not including KV cache or framework overhead. The table below gives rough magnitudes for common sizes:

Values are derived from parameter count × byte width; they exclude KV cache, activations, and framework overhead, and don't represent actual usage for any specific model.

Second Calculation: KV Cache and Its Multiplication with Context Length and Concurrency

The basic formula for KV cache per token is: 2 × layers × KV heads × head dim × bytes per parameter. Multiply this by sequence length and concurrency to get total KV cache usage. This per-token formula is the basic form from the vLLM official documentation. Context length increases KV cache linearly; doubling concurrency also doubles KV cache. FP16 vs FP8 matters here: FP8 KV cache saves 1 byte per parameter, significantly shrinking the cache. vLLM v0.27.0 extends FlashAttention 4 FP8 KV cache support on Blackwell SM100, meaning the same model with the same context can save substantial memory with FP8 cache.

Third and Fourth Calculations: Why Intermediate Tensors and Framework Overhead Must Be Reserved

Besides weights and KV cache, inference involves activation values, communication buffers, CUDA context, etc., plus memory fragmentation. vLLM sets gpu_memory_utilization to 0.9 by default, using only 90% of total GPU memory and reserving ~10% for fragmentation and runtime. For practical configuration of gpu_memory_utilization, see the full deployment guide at vLLM Deployment. This means "available memory" isn't the same as the card's nominal capacity. For example, a 24GB card actually has ~21.6GB usable. So, your theoretical usage must stay below this available value, or you risk CUDA out of memory.

Four-Step Calculation Walkthrough: From Model Scale to Target Memory Range

Here's how to choose GPU memory in practice:

1. Calculate weights: parameter count × bytes per parameter.
2. Calculate KV cache: per-token KV cache × context length × concurrency.
3. Account for intermediate tensors and framework overhead: reserve 10%-20% of total memory, or per engine docs.
4. Add up for theoretical minimum, then multiply by 1.1-1.2 for a practical range.

You'll get a range, not a single point. The lower bound is barely enough but risky; the upper bound is safer but may waste memory. For a 7B FP16 model with 8K context and concurrency 4 (estimated with 32 layers, 8 KV heads, head dim 128, GQA structure): weights ~14GB + KV cache ~4GB + intermediate and overhead ~3GB = ~21GB. With headroom, the range is 24-28GB, so a 24GB card is borderline (need to reduce concurrency or context); for stable production, 32GB or more is suggested. Note: if using a 32-head MHA structure (KV heads 32 instead of 8), per-token KV cache goes from ~128KB to ~0.5MB, making KV cache ~16GB, total ~33GB, and with headroom, a 48GB tier or multi-GPU is advised. If concurrency grows further, consider multi-GPU parallelism; see GPU Cluster solutions.

Engine Versions Rewrite the Memory Bill: FP8 KV Cache and Startup Optimizations Change Headroom

This is key: the inference engine version directly alters memory usage. According to vLLM v0.27.0 official release notes (August 2026), the version introduces sequence parallelism and routing optimizations for DeepSeek-V4, saving 448 MiB by skipping empty c128 startup and reducing E2E TTFT by 3.4%. It also deepens FlashAttention 4 FP8 KV cache and headdim-256 support on Blackwell SM100. These optimizations mean the same model and hardware may use less memory after upgrading. However, 448 MiB is specific to DeepSeek-V4 in particular scenarios; don't extrapolate to general savings. So when estimating memory, you must know your engine version and leave extra headroom for version differences.

Mapping Memory Range to GPU Tiers: When Single GPU Suffices, When Multi-GPU Is Needed

After calculating the range, compare with GPU options. Common tiers: 24GB, 48GB, 80GB+. Signals for larger memory or multi-GPU: long context (e.g., 32K+), high concurrency (tens of requests), KV cache exceeding weight size. A typical 24GB single card is RTX 4090; see RTX 4090 Cloud Server specs, suitable for 7B-class models with short/medium context and low concurrency for testing or light production. If concurrency rises to 16, KV cache doubles, and 24GB will OOM. Solutions: reduce precision to FP8/INT4, shorten context, or go for 48GB+ tier or multi-GPU. The final step is aligning your calculated range to the nearest tier above the upper bound, not buying at the theoretical minimum.

Measuring Memory Peak with Pay-as-You-Go Resources: Metrics, Stress Testing, and Checklist

After theoretical calculations, validate with real workloads:

• Choose target context length and concurrency.
• Deploy the model on a pay-as-you-go GPU instance and run inference.
• Observe memory peak, fluctuations, and any CUDA out of memory.
• Record OOM boundaries—the context or concurrency thresholds where memory runs out.

NexGPU offers various GPU models and pay-as-you-go pricing; you can start small to validate, then gradually increase load to measure peak memory, avoiding over- or under-buying. This validation is low-cost and yields real data. Checklist: any OOM, memory utilization over 90%, latency meeting requirements.

Five Common Misconceptions: Weights Only, Ignoring Concurrency, Treating Theoretical as Ceiling

• Misconception 1: Only weights, ignoring KV cache. E.g., 7B FP16 weights are 14GB, thinking a 16GB card is enough, but long context causes OOM.
• Misconception 2: Single-request estimation, ignoring concurrency. Production concurrency multiplies KV cache.
• Misconception 3: No fragmentation headroom. Using nominal memory, ignoring the 10% framework reserve.
• Misconception 4: Ignoring engine version differences. Newer vLLM may use less memory, but older versions may require more; selection must tie to a version.
• Misconception 5: Precision switch changes only weights, not cache. Going from FP16 to FP8 requires changing KV cache precision too, or you won't save expected memory.

Follow the four-step method to get a range, then use NexGPU's pay-as-you-go instances with prebuilt model/application templates to run a stress test before committing to a long-term configuration.

FAQ

How much memory does a 7B model need?

For a 7B FP16 model with 8K context and concurrency 4 (assuming 32 layers, 8 KV heads, head dim 128, GQA): weights 14GB, KV cache ~1GB per 8K context (per token ~128KB), concurrency 4 gives ~4GB, plus ~3GB overhead, total ~21GB. With headroom, the range is 24-28GB; a 24GB card is borderline (reduce concurrency or context), but stable production suggests 32GB+. With 32-head MHA, KV cache is ~4x (about 16GB), total ~33GB, recommending 48GB or more.

How to calculate KV cache memory usage?

Using the formula 2 × layers × KV heads × head dim × bytes per parameter, then multiply by sequence length and concurrency. For example, 32 layers, 8 KV heads (GQA), head dim 128, FP16: per token ~128KB, 8K context ~1GB, concurrency 4 gives ~4GB. FP8 halves this. With 32-head MHA, per token ~0.5MB, about 4x GQA (8 KV heads), scaling accordingly.

What's the memory difference between FP16 and FP8?

For weights, FP16 uses 2 bytes per parameter, FP8 uses 1 byte—half. KV cache also halved. For 7B FP16: weights 14GB, FP8 7GB; KV cache drops from 4GB to ~2GB (8K context, concurrency 4, assuming 32 layers, 8 KV heads, head dim 128).

How much headroom should I leave?

At least 10%-20% for fragmentation and runtime. vLLM defaults to 90% memory usage. If load fluctuates, increase headroom to 20%-30%. Better to overestimate to avoid frequent OOM.

How to resolve CUDA out of memory?

First reduce concurrency or context length, then consider lowering KV cache precision (e.g., FP8) or reducing batch size. Also upgrade engine version (like vLLM v0.27.0) to leverage optimizations, or use a larger GPU. Use pay-as-you-go testing as recommended in DeepSeek Deployment to pinpoint bottlenecks.