Architecture Review: Deploying vLLM Disaggregated Inference via Elastic GPU Rental

As concurrency in multi-agent dialogue systems surges, many teams reviewing their backend architectures face the dilemma of volatile first-token latency and choppy streaming output. To optimize response speed without inflating hardware balance sheets, one team decided to deploy a multi-node disaggregated architecture through GPU rental. This elastic approach helped them cut service response times by nearly half.

The team's pain point was that when many users initiated conversations simultaneously, the prefill phase of new requests would hog compute, causing severe latency for ongoing decode requests. In a single-node deployment, this conflict was nearly impossible to reconcile. Last week, with the official integration of vLLM and TileRT, the team found a viable path to physically separate the two stages.

Hardware Selection for Disaggregated Inference: Why Elastic GPU Rental

Traditional single-node inference servers often struggle to balance high throughput with ultra-low latency. Once hit by concurrent traffic spikes, new and old requests compete for VRAM and compute cores inside the GPU, leading to a degraded experience for all users. In such scenarios, adopting a physically separated Prefill/Decode architecture, assigning tasks of different intensities to different hardware nodes, is widely recognized as an efficient solution.

For small to medium teams with limited budgets, monthly rental of multiple high-spec GPU servers is a heavy expense. Cloud compute based on elastic GPU rental allows them to scale nodes on demand during traffic surges and release resources during lulls. The team leveraged the NexGpu platform for hardware selection, allocating compute instances of various specs as needed to maintain financial flexibility during the architectural transition.

In this review, the team chose two types of compute nodes with different focuses. For prefill nodes, they rented servers with large VRAM to handle long-context throughput demands. For decode nodes, they selected GPU instances with high bandwidth and strong small-batch compute capabilities. This hardware division, combined with the newly released vLLM x TileRT solution, laid the foundation for performance tuning.

Deployment in Practice: Building a Two-Stage Pipeline with vLLM and TileRT

The core highlight of this system is its zero-invasiveness. The integration solution released by vLLM last week uses the KVConnectorBase_V1 interface of the vLLM V1 version, meaning the team could directly load the external TileRT-specific decoding engine without modifying any underlying vLLM source code.

The deployment steps are as follows:

• Pull the official Docker image ghcr.io/tile-ai/tilert:cu132-latest to ensure the base environment includes all necessary GPU operator libraries;
• Install the latest TileRT package inside the container and run the built-in weight conversion script to convert standard GLM-5 weights to the engine's optimized format;
• Configure vLLM's MultiConnector, and in the startup parameters, specify the NIXL transfer mode via --kv-transfer-config to enable cross-node KV Cache transfer;
• Start the main prefill service node as the traffic entry point and route decode node requests to the TileRT process.

During specific parameter configuration, the team encountered a bottleneck in cross-node network communication. Due to the massive real-time transfer volume of KV Cache data, ordinary Ethernet connections caused severe network latency, negating all the acceleration benefits of the distributed architecture. To address this, they utilized the InfiniBand high-bandwidth channel between nodes provided by NexGpu, enabling RDMA single-sided write transfer, which fully overlapped the transfer phase with the compute phase.

Routing and Sharding: Balancing High Throughput and Ultra-Low Latency

In traditional inference deployments, increasing batch size to pursue throughput causes a cliff-like drop in per-user token output speed. The biggest significance of introducing TileRT lies in its extreme decode optimization for Batch Size = 1. Through highly customized compiled operators, it compresses GLM-5's token-by-token decoding time to the hardware limit.

To balance overall service throughput metrics with core users' latency experience, the team set up a lightweight intelligent router at the very front of the inference cluster. This router performs policy-based routing based on the type and channel priority of incoming requests. For background batch analysis tasks that are not latency-sensitive, traffic is still directed to vLLM's native decoding pool, leveraging its high-concurrency aggregate throughput advantage.

Conversely, for user requests from real-time chat interfaces that demand high streaming output speed, after the prefill phase completes in vLLM, the KV Cache state is instantly synchronized to the TileRT decoding pool via the network channel. In the decoding pool, each TileRT instance processes only a single request at a time, squeezing out extreme per-token output speed.

Post-Optimization Performance Review

After two days of production testing, this disaggregated architecture deployment showed clear advantages. Review data showed that at peak concurrency, end-to-end token generation per second for core users increased by nearly 1.8 times, and ITL (inter-token latency) jitter caused by prefill compute contention basically disappeared.

Additionally, in terms of compute efficiency, the team also did the math. Previously, high-end servers that were idle for long periods to handle occasional spikes have now been replaced with flexible cloud compute. By deploying this architecture on NexGpu, the team can release some TileRT instances with one click during off-peak hours, keeping only a small number of base nodes running, and reduced the overall GPU rental bill by about 35%.

For teams developing next-generation generative AI applications, performance optimization in the era of large models is no longer limited to squeezing a single framework. Combining scheduling services skilled at macro-level control with acceleration engines focused on point breakthroughs, and leveraging elastic compute acquisition channels, is a viable path for small and medium teams to break through in the compute red ocean.