Multi-Card Inference First-Token Latency Spikes: Cross-Process Cache Troubleshooting in GPU Rental Clusters

During a stress test of a multi-turn agent system last week, the first-token latency (TTFT) metric on the ops dashboard showed an abnormally steep upward curve. Originally averaging under 400 milliseconds, responses spiked to 3.98 seconds after the user conversation entered the third round, with P99 tail latency even exceeding 13 seconds.

For teams using GPU rental services, such latency not only ruins user experience but also means the rented GPU compute is not generating valid tokens—it is repeatedly recomputing the previous prompt (Prompt Prefill).

Why Does Multi-Card Concurrency Lead to KV Cache Invalidation?

The fault occurred on a single-node cluster composed of 8 H100 80GB GPUs. The team deployed an open-source large model service with FP8 precision and enabled conventional tensor parallelism.

After examining logs, the root cause was found in multi-process scheduling and cache isolation mechanisms. Traditional inference frameworks default to in-process cache. When a user sends the first round of requests, the system assigns that request to GPU process 3 (Rank 3), and the generated Key-Value cache (KV Cache) is stored in Rank 3's GPU memory.

However, when the same user sends the second round of messages a few seconds later, the load balancer randomly distributes the request to GPU process 6 (Rank 6). Since Rank 6's memory space is completely isolated from Rank 3, it cannot read previous historical caches.

As a result, Rank 6 must perform matrix multiplication on the entire historical context (tens of thousands of characters) again. As conversation rounds increase and concurrency rises, GPUs frequently enter full-load prefill computation, causing new requests to queue and first-token latency to skyrocket.

In vLLM's official roadmap released in July, high-interaction token scheduling and cache reuse for agent scenarios were listed as key areas to address. Without enabling cache sharing across processes, blindly adding more GPUs cannot solve response stuttering.

Cross-Process Sharing Mechanism: From Process Isolation to Unified Memory Pool

To thoroughly solve the bottleneck of repeated prefill computation, the team referenced the latest open-source shared cache solution (LMCache 0.4.3 architecture). The core idea is to decouple cache management from individual inference processes and build a cross-process shared KV cache pool at the host level.

On NexGpu's 8-card H100 rental node, each physical machine is equipped with ample GPU memory, large-capacity system memory (DRAM), and high-speed NVMe storage.

After the architecture adjustment, all GPU inference processes no longer work in isolation but register with an independent cache service on the host. After Rank 3 completes the first round of computation, the generated KV cache blocks are synchronously written to the host's shared memory pool.

When Rank 6 receives the second round of requests, it first queries the shared cache based on the token sequence's hash value. On a hit, it pulls the previous context to local GPU memory via PCIe, bypassing time-consuming prefill computation.

In a GPU rental environment, this architecture significantly reduces the hard dependency on single-card GPU memory capacity, freeing up memory that was previously idle or redundantly occupied for the actual decode phase.

From Configuration to Deployment: Concrete Steps for Shared Cache

On the operational level, troubleshooting this issue involves three steps: configuring shared memory space, starting an independent cache daemon, and adjusting inference engine parameters.

First, expand the shared memory mount between containers. In Docker or Kubernetes deployment manifests, increase the shared memory volume /dev/shm to over 64GB to ensure inter-process communication does not crash due to insufficient memory.

Second, start an independent cache daemon on the node. Set the maximum host memory usage by configuring the configuration file:

# 启动独立的跨进程缓存服务
python3 -m lmcache.server \
  --host 127.0.0.1 \
  --port 8080 \
  --max-memory-gb 200 \
  --device cpu

Finally, update the vLLM startup command to enable prefix caching and specify the external cache backend. To avoid GPU memory overflow, reasonably reduce the proportion of memory reserved by inference processes:

# 修改后的 vLLM 启动参数
vllm serve "Qwen/Qwen2.5-72B-Instruct" \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --kv-connector lmcache

Additionally, the team added tool call format standardization logic at the frontend routing layer to ensure consistent JSON ordering of identical system prompts and tool calls, preventing cache hash invalidation due to whitespace or key-value ordering differences.

Performance and Cost Benefits After Tuning in GPU Rental Clusters

After the architecture modification and successful validation, all metrics on the monitoring dashboard improved significantly.

On the same 8-card H100 node, average first-token latency for multi-turn conversations dropped from 3.98 seconds to 0.29 seconds, a ~13x reduction; P99 tail latency decreased from 13.55 seconds to 1.30 seconds. Meanwhile, the system's average decode throughput increased from 9.8 tokens per second to 37.4 tokens per second.

This performance improvement directly translates into tangible cost savings for GPU rental. By skipping redundant prefill computations, a single node can now support nearly 3x the number of concurrent sessions.

When deploying production-grade AI services, encountering latency spikes doesn't always require upgrading hardware specs. By clarifying the cache flow path between GPU memory and system memory, and replacing traditional siloed architecture with cross-process sharing mechanisms, you can unlock higher throughput efficiency under existing GPU resources.