Choosing Between Tensor Parallelism and Pipeline Parallelism: First Check If the Model Fits on a Single GPU, Then Look at Inter-GPU Interconnect

2026-10-10 608

In most cases, follow this rule: For multiple GPUs within a single machine that are connected via high-speed interconnects like NVLink, prioritize Tensor Parallelism (TP). When the model is too large to fit on one machine and requires cross-machine deployment, use TP within each machine and Pipeline Parallelism (PP) between machines. If the model and KV cache can fit on a single GPU, don't split the model. In this case, using a single GPU or running several independent replicas is usually simpler.

Below, we explain the reasoning behind this rule and several situations that require separate consideration.

Answer Three Questions First

Before you start, confirm three things:

  1. Is there enough VRAM: Can a single GPU hold the model weights plus KV cache at the precision you plan to use (BF16, FP8, INT4, etc.) and context length? The VRAM requirements vary greatly between models. You can first check the GPU selection guide for your model, or refer to How large a model can 48GB VRAM run.
  2. How are the GPUs connected: NVLink/NVSwitch or regular PCIe? If cross-machine, what network between nodes? Run nvidia-smi topo -m on the machine to see the interconnect type.
  3. What does the task prioritize: Online serving cares more about time-to-first-token and token generation speed; offline batch inference cares more about total throughput; training and fine-tuning follow a different approach, discussed separately later.

Once you have answers to these three questions, the choice is basically determined.

The Difference Between the Two Parallelisms

Tensor parallelism splits weights within each layer and synchronizes frequently; pipeline parallelism segments by layers, communicates only at stage boundaries, and has bubbles

Tensor parallelism splits within layers. In each Transformer layer, the attention QKV matrices and MLP weight matrices are split across multiple GPUs and computed simultaneously. Every GPU participates in every layer's computation, and after each layer, All-Reduce or All-Gather is needed to synchronize results.

Pipeline parallelism splits by layers. Consecutive layers are divided into stages. For example, layers 1–16 on GPU 0, layers 17–32 on GPU 1. GPUs only pass intermediate activations once at stage boundaries, using point-to-point communication.

The trade-offs of both stem from this difference:

Tensor Parallelism (TP)Pipeline Parallelism (PP)
Communication frequencySynchronization every layer, many timesTransfer only at stage boundaries
Interconnect requirementHigh bandwidth, low latency; usually requires NVLinkLower; can use cross-node network
Single-request latencyCan be reduced; multiple GPUs compute one token simultaneouslyIncreased; requests must go through stages sequentially
Main drawbackWith slow interconnect, communication overhead can offset compute gainsPipeline bubbles; GPUs idle when concurrency is low

Inference: How to Configure for Single and Multi-Machine

Single Machine, Multiple GPUs

When the model doesn't fit on a single GPU, first use TP within one machine. For example, with vLLM:

vllm serve <模型名或路径> --tensor-parallel-size 4

tensor_parallel_size is usually set to 2, 4, or 8, and must divide the model's number of attention heads. If it doesn't divide evenly, startup will fail.

The benefit of TP is that the computation for one token is distributed across multiple GPUs simultaneously, reducing both time-to-first-token (TTFT) and time-per-output-token (TPOT). Therefore, latency-sensitive online serving typically prefers TP.

Even 8 GPUs in a Single Machine Can't Fit

For very large models, full precision, or very long contexts, even 8 GPUs may not be enough. In this case, use hybrid parallelism: TP within a machine, with size equal to the number of GPUs per machine; PP between machines, with size equal to the number of nodes. For example, two 8-GPU machines:

vllm serve <模型名或路径> \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2

This arrangement keeps the most communication-intensive TP on intra-node high-speed interconnects, while the cross-machine network only handles the lighter PP transfers. If TP were across machines, each layer's synchronization would go over the network, significantly slowing things down.

Only PCIe Between GPUs

Many consumer GPUs and some PCIe versions of data center GPUs lack NVLink. In such environments, using large TP sizes may cause layer synchronization overhead to offset the benefits of multiple GPUs. Here's how to handle it:

  • If quantization can fit the model into fewer GPUs, reduce the number of GPUs first.
  • When multiple GPUs are truly needed, use combinations like TP=2 and PP=2, run both under real concurrency, and compare latency and throughput empirically rather than relying on theoretical estimates.
  • If multi-GPU throughput is far below expectations, refer to Why multi-GPU inference throughput doesn't double for troubleshooting.

Throughput Not High After Using PP

Pipeline parallelism has an inherent issue: if the previous stage hasn't finished, the next stage must wait. This idle time is called a pipeline bubble. With few requests, most GPUs will be waiting. To improve throughput, you need sufficient concurrent requests or techniques like micro-batching or chunked prefill to fill the pipeline. Therefore, PP is better suited for high-concurrency serving and offline batch inference, not for scenarios with only one or two users requiring fast responses. If your main issue is latency, see How to optimize high inference latency for large models to distinguish between slow first token and slow generation.

Training and Fine-Tuning: Don't Rush to Split the Model

For training, the order of choices differs from inference:

  1. When VRAM is sufficient, start with data parallelism and memory optimization, such as DDP, ZeRO, or FSDP. They don't split the model's computation, are simple to implement, and scale efficiently.
  2. When a single layer's parameters or activations exceed single-GPU capacity, add TP within the machine.
  3. When scaling to multiple machines, use the so-called 3D parallelism: limit TP to within 8 GPUs per machine to leverage NVLink; combine PP and DP between nodes to reduce cross-machine communication bottlenecks.

For most LoRA or full-parameter fine-tuning of small to medium models, the first step is sufficient.

Quick Reference

  • Fits on a single GPU: don't split the model; run multiple replicas for more throughput.
  • Fits on a single machine with NVLink: use TP only.
  • Fits on a single machine but only PCIe: minimize GPU count and empirically test both TP and PP combinations.
  • Doesn't fit on a single machine: TP within machine + PP across machines.
  • Training or fine-tuning: start with DP/ZeRO/FSDP; add TP when a single layer doesn't fit; use 3D parallelism for multiple machines.

Things to Confirm Before Renting GPUs

Once you've decided on the parallelism strategy, the hardware requirements become clear: TP requires enough GPUs in the same node, preferably with high-speed interconnect; PP across machines requires available network between nodes. On NexGPU, you can first filter machines with the right number of GPUs per node on the Pricing and Available Nodes page. After starting up, run nvidia-smi topo -m to confirm the interconnect type before deciding the TP size. If you have questions about node configuration, you can confirm with the bilingual English/Chinese support via Telegram. For deployment, you can directly use vLLM or TGI image templates without setting up the environment yourself.

A note on costs: When testing different parallelism combinations, if you merely stop the machine, storage fees still apply; billing completely stops only after destruction. Destroy configurations promptly after testing, and save any weights or test results you need to keep beforehand.

If after evaluation you determine that a cross-node multi-machine solution is needed, or you need a long-term cluster of fixed size, you can contact sales. Before reaching out, organize your model size, precision, and concurrency targets according to the Enterprise GPU Cluster Consultation Requirements Checklist to speed up the conversation.

Last updated on 2026-10-10 15:02:02

Related Posts

From Small-Scale Experiments to Large-Scale Production: Why Your AI Business ...
From Prototype to Production: How vLLM Deployment Breaks Through High-Concurr...
Tensor Parallelism vs Pipeline Parallelism: Three Steps to Decide Based on Me...
Why Doesn't Multi-GPU Inference Throughput Double? Troubleshoot in Three Step...
What's the Right Inference Parallelism? Start with the Minimum TP Based on VR...
Multi-GPU Inference GPU Utilization Won't Go Up? Locate the Real Bottleneck i...