In short, splitting KV Cache by sequence means dividing the KV cache of the same request along the token position (sequence length dimension) and storing it across multiple GPUs. For example, a 128K token request on 4 GPUs stores only about 32K positions' keys (K) and values (V) per GPU, instead of each GPU retaining the full 128K context.
In framework documentation and papers, this approach is generally called Context Parallelism (CP). If it is only split during the decoding phase, it is usually called Decode Context Parallelism (DCP).
It mainly solves one situation: when the context is too long for a single GPU to hold the KV cache, and splitting by attention heads (tensor parallelism) has already reached its limit. In this case, splitting by sequence can still distribute memory by adding more GPUs.
First, look at the dimensions of KV Cache
The KV Cache tensor generally has five dimensions: number of layers, batch size, number of KV heads, sequence length (seq_len), and head dimension. Talking about how to split KV Cache is essentially choosing one dimension to divide across multiple GPUs:
- Split by number of KV heads: the approach of tensor parallelism (TP)
- Split by layers: the approach of pipeline parallelism
- Split by seq_len: the sequence splitting discussed in this article
In conventional inference, even with tensor parallelism, the KV cache on each GPU covers all token positions of this request; TP only lets each GPU store fewer heads. As context grows longer, the KV cache on each GPU grows with the number of tokens.
The estimation method is straightforward: KV bytes per token = 2 (K and V) × number of layers × number of KV heads × head dimension × bytes per element.
Take a model with 80 layers, 8 KV heads, and head dimension 128, which is close to the common configuration of 70B-level GQA models. With FP16, the KV per token is about 320KB:
- A single 128K token request is about 40GB
- At 1M tokens, it is about 320GB, already exceeding the model's FP16 weights themselves (about 140GB)
So in scenarios from 128K to million-token contexts, KV Cache often takes more GPU memory than weights. For how to calculate specifically and what easier methods exist, see What to Do When Long-Context Inference Runs Out of GPU Memory.

How is it different from tensor parallelism?
Tensor parallelism splits along attention heads, and the problem lies in the structure of current mainstream models. GQA (Grouped Query Attention) and MQA reduce KV heads to very few, commonly only 8, and MQA even only 1. Splitting by heads, the number of splits is basically limited by the number of KV heads: 8 KV heads can be effectively split into at most 8 parts.
Adding more GPUs, the extra GPUs can usually only replicate KV heads, and the KV memory per GPU will not continue to decrease. The specific behavior depends on the framework used.
Splitting by sequence is not limited by the number of heads. The longer the sequence, the more parts it can be split into, and memory pressure can be distributed according to the number of GPUs. The two do not conflict; you can first use TP to split heads, then use CP to split the sequence. The actual combination depends on framework support. For how to choose between tensor parallelism and pipeline parallelism, refer to this comparison.
After splitting, how can attention still be computed correctly?
Attention requires each token to see all tokens before it, but the KV of these tokens is now distributed across different GPUs, so inter-GPU communication is necessary to fill the gap. The approaches differ between the Prefill and Decode phases.
Prefill phase: Split the long prompt into several segments, assign them to different GPUs to compute their own projections and local KV in parallel, then complete global attention using one of the following two methods:
- Ring Attention: Each GPU passes its KV block to the next GPU in a ring, computing while passing; after one full circle, each GPU has computed with all KV
- DeepSpeed-Ulysses: Use All-to-All communication to temporarily convert sequence splitting into head splitting, compute attention, then convert back
This way, the prefill of a long prompt can be shared by multiple GPUs, and time to first token (TTFT) will drop significantly.
Decode phase: Each time a new token is generated, its Query must compute attention with all historical KV. The approach is as follows:
- Each GPU uses its local segment of historical KV and this Query to compute a local attention result, while recording the LogSumExp (LSE) of this segment
- Collect the results from each GPU via communication
- Use LSE to correct and normalize, then merge
The denominator of softmax originally needs to sum over all positions; LSE exactly records the normalization amount for each segment, so after merging, the result is mathematically consistent with single-GPU computation, not an approximation. This way, each GPU does not need to replicate a full copy of the context.
It is not the same as PagedAttention
These two concepts are often confused:
- PagedAttention manages memory within a single GPU. It borrows from OS paging, placing KV Cache into fixed-size blocks to reduce fragmentation and waste, but it cannot solve the problem of insufficient total memory on one GPU.
- Sequence splitting is a cross-GPU, cross-node parallel strategy that solves exactly the problem of insufficient total memory on a single GPU.
Both can be used together. Inference frameworks such as vLLM and AWS Neuron build context parallelism on top of paged management.
The cost: communication is on the critical path of attention
Splitting by sequence is not free. Each layer of attention requires a cross-GPU communication, and this step cannot be skipped, so speed heavily depends on inter-GPU interconnect:
- On a single machine with multiple GPUs and high-speed interconnect like NVLink, the cost is relatively manageable
- With only PCIe, or across nodes over ordinary networks, communication time may eat most of the benefit
- When the context is not long, the memory and computation saved by splitting may not offset the communication overhead
For common reasons why speed does not keep up after adding GPUs, see Why Multi-GPU Inference Throughput Does Not Double. If your problem is mainly latency, first distinguish whether it is slow first token or slow token generation; refer to How to Optimize High LLM Inference Latency.
When is it worth considering?
When most of the following apply, sequence splitting is worth serious evaluation:
- The context of a single request is hundreds of thousands of tokens or more, and the KV Cache cannot fit on a single GPU
- Tensor parallelism is already split to the limit of the number of KV heads, and adding GPUs cannot reduce memory further
- There is high-speed inter-GPU interconnect, or you are willing to choose such nodes for it
- The TTFT of long prompts is the main pain point, and multiple GPUs are needed to do Prefill together
If the context is only tens of thousands of tokens, or memory is only slightly insufficient, it is usually more cost-effective to first try lower-cost methods: KV Cache quantization, limiting maximum context length, reducing concurrency. These methods do not require additional GPUs and do not introduce cross-GPU communication.
How to do it on rented GPUs
- Estimate first, then choose GPUs: Use the formula above to calculate the total KV Cache for the target context, add weights, and determine how many GPUs are needed and how much memory each GPU requires.
- Choose multi-GPU nodes with high-speed interconnect: Check the configuration of multi-GPU nodes on the Pricing and Rentable Nodes page, focusing on the inter-GPU interconnect method, not just the total memory.
- Choose an inference image and verify the version: On the Image Templates page, select a vLLM or TGI image. Different engines and versions have different levels of support for CP/DCP; parameter names and whether they are enabled by default are subject to the documentation of the corresponding version.
- Verify in two steps: First run with conventional tensor parallelism, record memory usage, TTFT, and token generation speed; then enable context parallelism and compare with the same long prompt. If there is clear improvement, apply it to the production configuration.
Multi-GPU debugging is billed by the hour. After comparison testing, handle unused instances promptly: storage fees still apply after shutdown, and all billing stops only after destruction. Models and data you need to keep should be saved before destruction.
NexGPU-算力租赁,GPU服务器,GPU云算力,AI服务器租用-新闻博客