Skip to main content
NexGPU

Inference runtime

Get vLLM running: the VRAM math, the parallelism, and the card that actually fits

vLLM is not a model. It is the engine that turns a model into a high-concurrency service. Before you install it, you only need to settle one question: does this card hold the weights plus the KV cache?

Sign up freeView pricing

vLLM came out of the Sky Computing Lab at UC Berkeley and is now maintained by more than two thousand contributors under Apache-2.0, with close to 90k stars on GitHub. What it does is specific: PagedAttention manages the KV cache in pages, continuous batching lets new requests slot into a batch already in flight, and on top of that sit chunked prefill, prefix caching, CUDA graphs and speculative decoding. The result is that the same card serves an order of magnitude more throughput than a naive transformers loop, which is why almost every team that wants to put its own model behind an endpoint ends up back at vLLM.

What actually stops people is not performance, it is VRAM. On startup vLLM claims the card at gpu_memory_utilization=0.92 by default, loads the weights, and hands everything left over to the KV cache. So there is only one equation that matters: weight bytes = parameter count x bytes per parameter (bf16 is 2 bytes, FP8/INT8 is 1, 4-bit is roughly 0.5 plus quantization scale overhead). Then you still need room for the KV cache, or max_model_len has to come down. A 70B model in bf16 is 140GB and no flag changes that; the same model at 4-bit is roughly 35-40GB and fits on a single 48GB card. That gap is what decides which GPU you should rent.

There is also a hard floor most people discover on their first install: the official prebuilt wheels require NVIDIA compute capability 7.5 or higher - the T4, RTX 20-series, A100, L4, H100, B200 tier. That puts the Tesla V100 (7.0) and Tesla P40 (6.1) outside official support. Architecture matters further up the stack too: llm-compressor FP8 W8A8 is available only on Ada, Hopper and AMD cards, so Ampere parts like the A100, A6000 and 3090 cannot take that path and have to run bf16 or 4-bit instead. None of this is folklore - it is what you need to know before you pick a card.

01 —

vLLM release lines and install variants

Stable for production, the preview channel for early validation. The three columns below cover coverage, VRAM-related defaults, and context/KV changes.

VersionParametersVRAMContextNotes
v0.27.1 (current stable)200+ HuggingFace architectures: dense, MoE, hybrid state-space, multimodal, embedding and reward modelsgpu_memory_utilization defaults to 0.92, kv-cache-dtype to auto, dtype to automax-model-len is derived from the model config when left unsetThe patch release on top of v0.27.0 - pin this for production. Alongside the OpenAI-compatible server it also exposes an Anthropic Messages API and gRPC.
v0.27.0Adds full-stack Kimi K3 support (kernels, Python and Rust frontends), Qwen3.5 dense and MoE, K-EXAONE-2.0-750B-A37B, jina-embeddings-v5-text-nanoQuantization gains FP4 Qutlass for compressed-tensors, MXFP8 linear in INC, AutoRound W4A16 MoE, and a TurboQuant KV quant modeFlashAttention 4 on SM100 supports FP8 KV cache and headdim-256561 commits from 242 contributors. DeepSeek-V4 kernels roughly 2x faster; dependencies jump to PyTorch 2.13.0, Triton 3.7.1, Transformers 5.14.1, FlashInfer 0.6.16.post3 and NCCL 2.30.7 - check your image before upgrading. Breaking: max_num_partial_prefills and max_long_partial_prefills were removed, and Plamo2 and Ouro were dropped.
v0.26.0Introduces the Inkling model family with llm-compressor NVFP4 weights and compressed-tensors dynamic FP8KV offloading matures: generic P2P secondary tiers, pluggable eviction policies, parallelism-agnostic per-layer mappingsAttention backend can be selected per attention groupGeneration models can run lm_head in fp32 via head_dtype for steadier tail sampling. When VRAM is tight, tiered KV offload is the most useful thing this release shipped.
v0.28.0rc1 / rc2 (preview channel)Early validation for next-generation models and kernelsInherits the stable defaults-Not a final release: APIs and flags can still move. Good for checking whether a new model runs on a per-second instance, not for production.
Prebuilt wheel variants (cu128 / cu129 / cu130)Python 3.10-3.13Minimum NVIDIA compute capability 7.5 (T4, RTX 20-series, A100, L4, H100, B200)Default wheels bundle CUDA 12.9, with CUDA 12.8 and 13.0 builds also published`uv pip install vllm --torch-backend=auto` picks the wheel from your driver. Beyond NVIDIA there are plugin backends for ROCm, CPU (x86/ARM/PowerPC), TPU, Gaudi and Apple Silicon.

02 —

Once the math is done, here is the card to rent

Weights = parameter count x bytes per parameter, and whatever is left goes to the KV cache. Every tier below was chosen from that equation.

  • 7B-14B in bf16, or 32B at 4-bit, single card for integration and self-testing

    RTX 4090 24GB$0.540/GPU-hour

    Ada architecture, so Marlin kernels and llm-compressor FP8 W8A8 are both available. 8B bf16 weights are ~16GB and 32B AWQ 4-bit ~18GB, leaving real headroom for the KV cache inside 24GB.

  • 70B at 4-bit running resident on one card, or 32B bf16 with a long context

    RTX A6000 48GB$0.817/GPU-hour

    70B quantized to 4-bit lands around 35-40GB of weights, which fits in 48GB with room to spare - no tensor parallelism, and none of the NCCL, /dev/shm and IPC_LOCK debugging that comes with it.

  • 70B bf16 in production across two cards with tensor_parallel_size=2

    A100 SXM4 80GB$1.088/GPU-hour

    140GB of weights fits two 80GB cards exactly, and the SXM4 NVLink keeps the TP all-reduce off the critical path. Note that Ampere cannot use llm-compressor FP8 W8A8, so this tier stays on bf16.

  • Large MoE models like DeepSeek-V4 and Kimi K3, native FP8 and very long context

    H200 141GB$6.660/GPU-hour

    Hopper supports FP8 natively and 141GB per card means fewer shards to cut. For MoE remember enable_expert_parallel=True - expert parallel degree matches tensor parallel degree.

03 —

Four steps to a running vLLM service

From an empty instance to something the OpenAI SDK can call directly, with nothing skipped in between

  1. 01

    Start the instance and install vLLM

    Pick a card in the NexGPU console that satisfies the equation above. The 2,000+ prebuilt images already include PyTorch and vLLM environments. To build from a clean box, the project recommends uv, and --torch-backend=auto selects the cu128/cu129/cu130 wheel from your driver.

    uv venv --python 3.12 --seed && source .venv/bin/activate && uv pip install vllm --torch-backend=auto
  2. 02

    Launch the OpenAI-compatible server

    vllm serve turns the model into an HTTP service on http://localhost:8000. When VRAM is tight, cut max-model-len first - it is the single most effective lever. Only when one card genuinely cannot hold the weights do you reach for tensor-parallel-size, usually set to the number of GPUs.

    vllm serve <your-model> --tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.92
  3. 03

    Verify with a client you already have

    The routes are /v1/completions and /v1/chat/completions, and api_key can simply be EMPTY. Existing OpenAI SDK code moves over by changing base_url alone, which is why this is the shortest migration path from a closed API to self-hosting.

    curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model":"<your-model>","prompt":"San Francisco is a","max_tokens":7,"temperature":0}'
  4. 04

    Tune throughput, handle preemption and OOM

    Preemption warnings in the log mean the KV cache is short - vLLM preempts by RECOMPUTE rather than SWAP by default. Lower max-num-seqs or max-num-batched-tokens, or raise the parallel degree. Chunked prefill is on by default; a small max-num-batched-tokens (2048) improves inter-token latency while values above 8192 favour throughput. If startup drags, trade compile time with -O0 through -O3, reuse the compile cache via VLLM_CACHE_ROOT, or skip memory profiling with --kv-cache-memory.

    vllm serve <your-model> --max-num-batched-tokens 8192 --max-num-seqs 64 --enable-prefix-caching

The arithmetic, in full

Start with the debugging phase: a single RTX 4090 24GB is $0.540/GPU-hour, so eight hours a day for five days is 40 x $0.540 = $21.60 to get a 7B-14B service fully wired up. For a 70B bf16 production endpoint, two A100 SXM4 80GB at tensor_parallel_size=2 cost 2 x $1.088 = $2.176/hour; resident for a full 720-hour month that is 2.176 x 720 = $1,566.72. If the weights stay on disk, 140GB at the median storage rate of $0.414/GB-month is 140 x 0.414 = $57.96/month - and storage bills separately from compute, so the moment the instance stops, the $2.176/hour stops with it and only the $57.96 disk keeps counting until you destroy it. Want a throughput run on an H200 141GB before launch? Two hours is 2 x $6.660 = $13.32. Metered per second, no minimum, no setup fee, no quota request - shut it down when the test ends.

04 —

FAQ

How much VRAM does vLLM actually need? Is there a formula I can just apply?

Yes: weight bytes = parameter count x bytes per parameter. bf16/fp16 is 2 bytes, FP8 or INT8 is 1, and 4-bit quantization is roughly 0.5 plus quantization scale overhead. So 8B bf16 is ~16GB, 32B bf16 ~64GB, 70B bf16 ~140GB - and that same 70B at 4-bit is ~35-40GB. After the weights you still need the KV cache: vLLM claims the card at gpu_memory_utilization=0.92 and gives everything past the weights to KV, so when KV runs short, max-model-len has to come down. If you are unsure, rent a card by the second on NexGPU and measure it once - that beats any estimate table and costs pocket change.

Which GPUs does vLLM support? Can my V100 or P40 still run it?

The official prebuilt wheels require NVIDIA compute capability 7.5 or higher, and the docs name the T4, RTX 20-series, A100, L4, H100 and B200 tier. The Tesla V100 is 7.0 and the Tesla P40 is 6.1, both below that line and outside official support. Architecture differences matter too: Marlin needs Turing or newer and Turing does not support Marlin MXFP4, while llm-compressor FP8 W8A8 is limited to Ada, Hopper and AMD - Ampere parts like the A100, A6000 and 3090 cannot use it. On NexGPU you can spin up anything from a Tesla T4 16GB ($0.298/GPU-hour) to an H200 141GB ($6.660/GPU-hour) rather than buying a card to test an architecture.

How is vLLM different from Ollama or llama.cpp, and which should I use?

They aim at different problems. llama.cpp and Ollama are strongest for one machine and one user, with GGUF quantization and workable CPU inference - great for local tinkering. vLLM is built for concurrent serving: PagedAttention for the KV cache, continuous batching so requests join a batch in flight, chunked prefill on by default, prefix caching to reuse shared prefixes. Under real concurrency the throughput is in a different class, and it ships an OpenAI-compatible API, an Anthropic Messages API and gRPC. The test is simple: if you are the only user, take the former; if you are serving a team or a product, take vLLM. To compare directly, run both on one RTX 4090 24GB on NexGPU - two hours is under $1.10.

It OOMs on startup, or the log keeps printing preemption warnings. What now?

In order of effect: cut max-model-len first, then lower max-num-seqs and max-num-batched-tokens, then consider raising tensor-parallel-size or pipeline-parallel-size. vLLM's default preemption mode is RECOMPUTE rather than SWAP, so a flood of preemption warnings simply means the KV cache is undersized. A few more memory levers: enforce_eager=True disables graph capture entirely, cudagraph_capture_sizes limits capture to a few batch sizes, and for multimodal work limit_mm_per_prompt and mm_processor_cache_gb (4 GiB by default) both help. If the card is simply too small, do not keep shaving context - move to an RTX A6000 48GB on NexGPU at $0.817/GPU-hour. Billed per second, switching cards costs far less than a night of parameter tuning.

How should I split across GPUs - tensor parallel or pipeline parallel?

The guidance is explicit: use tensor parallelism when the model does not fit on one GPU, then add pipeline parallelism once tensor parallelism is maxed out or when you need to cross nodes. Both tensor_parallel_size and pipeline_parallel_size default to 1. MoE models want enable_expert_parallel=True, with expert parallel degree matching tensor parallel degree; if VRAM is fine and you just need more request capacity, data_parallel_size=N to replicate the model beats splitting it. Most multi-GPU pain is in the communication layer: an ncclCommInitRank failure usually means the container lacks IPC_LOCK or /dev/shm is not mounted, and a wrong interface can be pinned with VLLM_HOST_IP and NCCL_SOCKET_IFNAME. NexGPU supports up to 14 GPUs per node with a max node VRAM of 2,152GB, so TP stays inside one box and half of these problems never appear.

Once I self-host vLLM, how much of my existing OpenAI API code has to change?

Essentially one line: point base_url at http://your-host:8000/v1 and set api_key to EMPTY. /v1/completions and /v1/chat/completions are both there, and the project additionally ships an Anthropic Messages API and gRPC, so migration costs less than most people expect. The time goes into picking the card and tuning throughput, not rewriting code. NexGPU spans 51 countries and regions with 1,175 verified rentable nodes, 2,498 GPUs and 75 GPU models, reachable over SSH, Jupyter, web terminal, REST API and CLI, with vLLM and PyTorch already in the 2,000+ prebuilt images and bilingual support on Telegram with no ticket queue.

More in Local runtimes

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.