vLLM Multi-GPU Tensor Parallel Configuration Guide: How to Set TP and 5-Step Troubleshooting

Conclusion First: Parameter Decision Order for vLLM Multi-GPU Tensor Parallel Configuration

Many people assume that increasing --tensor-parallel-size will always make multi-GPU inference faster, but in reality, the TP setting is determined by both memory allocation and model attention head count—bigger is not always better. The correct approach is: first calculate the TP lower bound based on four memory accounts, then determine the upper bound using attention head divisibility constraints and inter-GPU interconnect, then sequentially adjust --gpu-memory-utilization, --max-model-len, and KV Cache precision, and finally run comparative validation with identical request distribution. The official vLLM documentation states that for single-machine multi-GPU setups, tensor parallelism is preferred, and pipeline parallelism is only added across nodes to avoid slow network impacting high-frequency communication Official Parallelism Documentation. This article follows this order to provide you with a directly actionable vLLM multi-GPU tensor parallel configuration guide.

Step 1: Calculate TP Lower Bound with Four Memory Accounts

When starting vLLM, memory allocation follows a fixed order: first load model weights, then perform a dummy forward pass to probe intermediate activations, and finally allocate the remaining memory (default around 0.90-0.92) to the PagedAttention KV Cache pool according to --gpu-memory-utilization. Therefore, the minimum number of GPUs you need depends on four accounts: weights, KV Cache, intermediate activations, and engine constant overhead.

The estimation method is simple: first multiply the total single-GPU memory by --gpu-memory-utilization to get usable memory; then divide the model weight size plus the expected KV Cache usage (related to maximum context length and concurrency) by the usable single-GPU memory, and round up to get the TP lower bound. The actual usage of intermediate activations and engine constant overhead should be based on local profiling results. It is recommended to conservatively reserve headroom first, then adjust based on the KV Cache block count in actual startup logs. If available memory is close to the requirement, run adjacent TP values on the same model to decide based on startup logs and stress test results. For specific model weight and KV Cache magnitudes, refer to Qwen Deployment Memory Requirements.

Step 2: Attention Head Divisibility and Interconnect Determine TP Upper Bound

TP cannot be set arbitrarily. vLLM requires that the total number of Query heads and, under GQA architecture, the number of KV heads be divisible by --tensor-parallel-size, otherwise the initialization will throw a dimension splitting assertion error and the service won't start. For example, a model with 32 Query heads can be divided by 2, 4, 8, but not by 3, 5, 6, so TP=3/5/6 is unavailable on most mainstream architectures. Before deployment, verify with the num_attention_heads and num_key_value_heads in your model config.

Another upper bound is inter-GPU interconnect bandwidth. Tensor parallelism relies on high-speed buses (like NVLink) for operator-level splitting and All-Reduce communication. If you use PCIe-connected GPUs, cross-GPU communication overhead will significantly degrade performance, so TP should not be set too high. Therefore, TP upper bound = min(max value satisfying head divisibility, split count allowed by interconnect bandwidth). For more practical insights on cross-GPU communication and split benefits, refer to Multi-GPU Inference Optimization.

Step 3: How to Divide TP and PP, and When to Use Cross-Node Pipeline Parallelism

The division of labor between --tensor-parallel-size and --pipeline-parallel-size is clear: within a single machine, prioritize maximizing TP (TP = number of GPUs in the machine) because NVLink bandwidth is high; when the model exceeds the total memory of a single machine, add cross-node PP, usually PP = number of nodes. This combination avoids slow cross-node networks carrying high-frequency tensor communication.

A common misconfiguration is abusing PP within a single machine—when there is high-speed interconnect between GPUs, cutting into PP causes pipeline bubbles and reduces utilization. Another misconfiguration is forcing TP across nodes, making All-Reduce travel over slow networks, crashing throughput. So when crossing nodes, the correct approach is to fully split TP within each machine, then split PP across nodes.

It should be noted that for hundred-billion-scale MoE models, the 3D hybrid split ratio of Expert Parallelism with TP/PP across nodes is still evolving across hardware platforms. The TP/PP combination conclusions in this article only cover dense models in single-machine and multi-machine conventional topologies, and do not provide a definitive optimal split for MoE.

Example startup command:

# 单机 4 卡，TP=4
vllm serve your-model --tensor-parallel-size 4

# 跨 2 节点，每节点 4 卡，TP=4，PP=2
vllm serve your-model --tensor-parallel-size 4 --pipeline-parallel-size 2

Step 4: Adjustment Order for gpu-memory-utilization and max-model-len

Both parameters modify the same KV Cache pool, so they must be adjusted in order, one at a time. --gpu-memory-utilization determines the pool size (default 0.90-0.92), while --max-model-len determines the logical context length reserved per request. If you set --max-model-len to the model's theoretical maximum, a single request will consume excessive KV blocks, drastically reducing system concurrency NVIDIA Blog.

The correct order is: first fix --gpu-memory-utilization, then adjust --max-model-len based on actual request length, observing throughput and concurrency changes. Changing both parameters simultaneously makes it impossible to diagnose issues. It is generally recommended to set --max-model-len slightly larger than your longest real request, not the model limit.

Step 5: How to Enable vLLM FP8 KV Cache and Where to Allocate Freed Headroom

When KV Cache becomes a bottleneck, you can enable FP8 quantization: --kv-cache-dtype fp8 (or fp8_e4m3 / fp8_e5m2 if hardware supports). This does not quantize model weights; it only reduces KV Cache memory per token by approximately 50%, effectively doubling token capacity FP8 KV Cache Documentation.

How to allocate the freed headroom? It depends on the scenario: if your business involves long documents or long contexts, prioritize increasing --max-model-len; if it's chat or high-concurrency APIs, prioritize increasing concurrency. Note that FP8 introduces precision loss, so quality regression tests are mandatory before deployment. If you lack a deep understanding of memory allocation, refer to GPU Cluster Planning for overall planning.

Why Engine Version Must Be Included in Configuration Baseline

With the same startup parameters, different vLLM engine versions may not reproduce the same results—because operator implementations, defaults, and memory profiling behavior may change. Therefore, the configuration baseline must record four items: engine version, model version, complete startup command, and hardware topology. After upgrading, rerun the same comparison to ensure performance changes are not due to version drift.

Post-Configuration Validation: TP=1/2/4 Comparison with Same Request Distribution

After configuring according to this guide, during validation fix the model, request distribution, and concurrency gradient, and record TTFT (time to first token), per-token latency, total throughput, max concurrency, and memory usage. Note: the main benefits of TP are reducing per-GPU memory and lowering TTFT, but cross-GPU All-Reduce overhead may cause total throughput to decrease, especially in Decode-heavy high-concurrency scenarios. So you must test empirically, not assume linear scaling.

Using NexGPU's pay-as-you-go multi-GPU instances, you can quickly spin up three environments with TP=1/2/4, using pre-built templates to ensure consistent engine versions and avoid drift.

Reversing the five steps of this guide gives you the troubleshooting order table below.

Troubleshooting Order Table

Frequently Asked Questions

What is the appropriate TP setting?

TP depends on model head divisibility constraints and memory requirements. First calculate memory to determine the lower bound, then check head divisibility for the upper bound, and take the maximum value satisfying both. For example, a 32-head model supports 2, 4, 8, but not 3, 5, 6.

What is the difference between TP and PP?

TP splits a single layer across multiple GPUs for parallel computation, suitable for single-machine high-speed interconnect; PP splits by layers, suitable for cross-node. Prefer TP within a machine, and use PP only across nodes to avoid slow network bottlenecks.

Why is dual-GPU throughput not improving?

The most common reason is not parameters but interconnect: TP benefits mainly come from reducing per-GPU memory and lowering TTFT. If two GPUs use PCIe instead of NVLink, or the scenario is Decode-heavy high concurrency, cross-GPU All-Reduce communication overhead may offset parallel gains, causing total throughput to stay flat or even drop. Additionally, not enabling --gpu-memory-utilization tuning may result in insufficient KV Cache, or improper TP settings introduce communication overhead, or --max-model-len set too high compresses concurrency. Check each item in the troubleshooting table.

What should gpu-memory-utilization be set to?

Default 0.90-0.92 is fine, but if OOM occurs, gradually lower to 0.8 or 0.85 while observing throughput changes. Do not set it too low, otherwise KV Cache becomes too small and affects concurrency.

How to enable FP8 KV Cache?

Add --kv-cache-dtype fp8 to the startup command, or use fp8_e4m3 / fp8_e5m2 if hardware supports. After enabling, KV Cache usage drops by about 50%, but quality regression tests are required.

What to do if startup reports CUDA out of memory?

First lower --gpu-memory-utilization to 0.8, then lower --max-model-len. If still insufficient, consider enabling FP8 KV Cache or increasing TP to reduce per-GPU memory.

It is recommended to first run a TP=1/2/4 comparison on your own model and real request distribution according to the five steps above, saving engine version, startup command, hardware topology, and measured metrics as a configuration baseline. After upgrading the engine, rerun the same comparison. If you need temporary multi-GPU environments for this comparison, use NexGPU's pay-as-you-go multi-GPU instances with pre-built templates to launch the same engine version, avoiding version drift that makes parameters irreproducible.