Inference Runtime

Self-hosting TensorRT LLM: what it actually costs in VRAM, and which card clears the bar

NVIDIA's own open-source LLM inference library, Apache 2.0. After 1.2 it is no longer the "compile an engine first" framework you remember — here is what it looks like today, what it eats, and which GPU to rent.

TensorRT-LLM · self-hosted

TensorRT LLM is maintained by NVIDIA at github.com/NVIDIA/TensorRT-LLM under Apache 2.0. The change that matters happened across the 1.0-to-1.2 line: the PyTorch-native backend went from optional, to default, to **the only one**. The 1.2 release notes say it plainly — `LLM(backend="tensorrt")` now raises a ValueError. Which means the large body of 2024-era tutorials that walk you through `trtllm-build`, converting a checkpoint into a `.engine` and loading it, will fail at step one today. That is the single biggest trap when people search for how to self-host TensorRT-LLM. The current stable line is 1.2; main has moved on to 1.3.0rc.

TensorRT LLM is not a model, so it has no fixed VRAM footprint. Three things add up: weights + KV cache + activation peak. Most first-time OOMs are not the weights — they are the KV cache. `free_gpu_memory_fraction` defaults to 0.9, meaning the server claims 90% of free VRAM for the KV pool the moment it starts. An 8B model in FP8 is only about 9GB and fits a 24GB card with room to spare, yet under defaults the card is at the ceiling before you send a single request. Dropping that to 0.80–0.85 and capping `--max_num_tokens` is the first thing to do on every deployment. Block reuse (`enable_block_reuse`) and partial reuse are on by default, which saves a real chunk on multi-turn chat and shared-prefix workloads.

The other hard constraint is that quantization formats are welded to GPU generations, and that — not raw VRAM — should drive your card choice. Ampere (A100, A6000, 3090) has no FP8 tensor cores, so you run W4A16 AWQ/GPTQ or plain BF16. Ada (4090) and Hopper (H100) give you FP8. NVFP4 and MXFP4 need Blackwell, with sm120 (RTX 5090) covering NVFP4, MXFP4 and FP8 per-tensor. At the system level, do not expect flexibility: Linux x86_64 and aarch64 only, no Windows, and the current support matrix lists neither Turing nor Volta — NVIDIA states outright that architectures not on the list are neither developed against nor tested. On NexGPU the cheapest supported card also happens to be the cheapest card, full stop: RTX 3090 24GB at $0.193/GPU-hr, billed per second.

01 —

Release lines: check which generation your tutorial is from

TensorRT LLM broke its own API across the 1.x line. Wrong version, wrong commands, all of them.

Version	Parameters	VRAM	Context	Notes
TensorRT LLM 1.3.0rc (main)	PyTorch-native backend + AutoDeploy beta	NVFP4 / MXFP4 / NVFP4 KV Cache, Blackwell only	CUDA 13.1 · PyTorch 2.10 · Python 3.12 · Ubuntu 24.04	Bleeding edge. New speculative decoding methods — DFlash, PARD, Suffix Automaton — land here first. The API can shift under you; do not make an rc your production baseline.
TensorRT LLM 1.2 (current stable)	PyTorch as sole execution backend	FP8 weights fit a 70B into a single 80GB card	Linux x86_64 / aarch64, no Windows	This is where the breaking changes live: the TRT backend is gone and `LLM(backend="tensorrt")` raises ValueError; CLI flags now override YAML config values. Start new projects here.
TensorRT LLM 1.1	Last release with both backends present	FP8 KV Cache available, another cut in footprint	Same dependency set as 1.0	Added GPT-OSS and Hunyuan support plus the KV Cache Connector API that disaggregated serving is built on. The C++ TRTLLM sampler became the default, with a new `sampler_type` argument.
TensorRT LLM 1.0	LLM API declared stable	Same as 1.1	PyTorch architecture set as the default backend	The watershed release. Runtime initialization moved out of the first call and into `__init__`, so your first generate() no longer stalls — the cost just moves to the constructor.
TensorRT LLM 0.21 and earlier	trtllm-build engine compilation flow	Extra VRAM spike during engine build	TensorRT 10.x dependency	The legacy path: convert checkpoint → `trtllm-build` → `.engine` → load. Removed entirely in 1.2. Almost every blog post still online describes this generation; the tell is the presence of `trtllm-build`.

02 —

Picking a card: the quantization format decides the generation

FP8 needs Ada or newer, NVFP4 needs Blackwell, Ampere is INT4 only — work backwards from the precision you need.

8B-class FP8 on one card, getting the pipeline working
RTX 4090 24GB$0.540/GPU-hr
Ada (sm89) is the cheapest tier with real FP8 tensor cores; nvidia/Qwen3-8B-FP8 is about 9GB of weights, leaving ~15GB of KV pool for genuine concurrency.
20B–32B on the newer MXFP4 / NVFP4 formats
RTX 5090 32GB$0.723/GPU-hr
sm120 is the consumer Blackwell that supports NVFP4, MXFP4 and FP8 KV Cache — natively MXFP4 weights like GPT-OSS run on their intended path here, with no dequant detour.
70B-class W4A16 AWQ with long context
A100 PCIE 80GB$0.824/GPU-hr
Ampere has no FP8, so 70B means INT4 AWQ/GPTQ at roughly 40GB of weights — and 80GB leaves exactly the headroom a long-context KV cache wants.
Production FP8 throughput, large-MoE tensor parallel, disaggregated serving
H100 SXM 80GB$3.582/GPU-hr
The full FP8 story — block scaling and rowwise — only exists on Hopper, and NVLink keeps `--tp_size 2/4` and prefill-decode KV transfer from being bottlenecked by the interconnect.

03 —

Four steps to a running trtllm-serve

Go through the official container and skip the CUDA 13.1 / PyTorch 2.10 version hell entirely.

01
Spin up a card and pull the NGC container
Do not pip-install bare metal on your first attempt. The official release image on nvcr.io already has CUDA, PyTorch and OpenMPI aligned. The `--ipc=host` flag and both ulimits are required by NVIDIA's own docs — omit them and you will fall over on multi-process work and large KV pools. If you insist on pip, the order matters: `pip3 install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu130`, then `apt-get install libopenmpi-dev`, then `pip3 install tensorrt_llm`.
```
docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus=all nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24
```
02
Start the OpenAI-compatible server, and rein in the KV cache first
`trtllm-serve` takes a HuggingFace model ID directly — no precompilation of any kind. Note the explicit `--kv_cache_free_gpu_memory_fraction 0.85`: the 0.9 default will eat nearly all free VRAM and is the number one cause of first-run OOM. `--max_num_tokens` caps the per-batch token budget, and `--backend pytorch` is technically the only option after 1.2 — spelled out here so the config documents itself.
```
trtllm-serve serve nvidia/Qwen3-8B-FP8 --tp_size 1 --max_batch_size 32 --max_num_tokens 8192 --kv_cache_free_gpu_memory_fraction 0.85 --backend pytorch --host 0.0.0.0 --port 8000
```
03
Verify it really is just an OpenAI endpoint
Once up you get standard `/v1/chat/completions` and `/v1/completions`; any OpenAI SDK works by changing base_url. The point of this step is confirming that weights, tokenizer and chat template all line up — if output is garbled or never stops, the chat template almost certainly did not match, so pass `--chat_template` explicitly.
```
curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "nvidia/Qwen3-8B-FP8", "messages":[{"role": "user", "content": "Introduce yourself in one sentence."}], "max_tokens": 64, "temperature": 0}'
```
04
Benchmark real capacity with trtllm-bench before sizing the fleet
This is where TensorRT LLM earns its keep over other runtimes: the benchmarking harness ships with it. Synthesize a dataset shaped like your actual traffic, then measure throughput. Set `--input-mean` and `--output-mean` to your real prompt and completion lengths — otherwise the TTFT, TPOT and tokens/s numbers mean nothing. Feed the `--report_json` output into a simple "QPS per card" figure and you know how many cards to rent instead of guessing.
```
trtllm-bench --model nvidia/Qwen3-8B-FP8 throughput --dataset /tmp/synth.jsonl --max_batch_size 32 --concurrency 16 --streaming --report_json /tmp/report.json
```

What one full validation round actually costs

Take Qwen3-8B-FP8 on an RTX 4090 24GB. Boot, pull the nvcr.io image, download weights, start `trtllm-serve` and finish warmup — call it 30 minutes: 0.5 hr × $0.540 = $0.27. Then run `trtllm-bench` at three concurrency levels (8 / 16 / 32) for 40 minutes each, 2 hours total: 2 × $0.540 = $1.08. The round comes to $1.35, and what you get back is the real TTFT, TPOT and throughput curve on that card rather than someone else's blog numbers. One tier up: 70B-class W4A16 AWQ on an A100 PCIE 80GB at $0.824/GPU-hr, a full 12-hour tuning day is 12 × $0.824 = $9.89. If the workload genuinely requires FP8 — which Ampere cannot do, so it has to be Hopper — an H100 SXM 80GB with `--tp_size 2` runs 2 × $3.582 = $7.164/hr, and a 6-hour benchmark sweep is 6 × $7.164 = $42.98. For perspective, the sticker price of one H100 buys you years of benchmarking here. Storage is billed separately. A 70B INT4 checkpoint is roughly 40GB: $0.414/GB-month × 40 = $16.56/month, about $0.55 a day. Compute is metered per second and stops when the instance stops, but storage keeps accruing until you destroy the volume — delete weight volumes you are done with. No minimum spend, no setup fee, no quota request.

04 —

FAQ

Do I still need trtllm-build to compile an engine? Why does every tutorial I follow error out?

You do not, and the option was removed outright. The PyTorch backend became the default in 1.0 and the only backend in 1.2, where `LLM(backend="tensorrt")` raises a ValueError — the `.engine` workflow no longer has an entry point. Your errors almost certainly come from a tutorial written against 0.21 or earlier. The correct move today is one command: `trtllm-serve serve <HF model ID>`. To confirm it for yourself, rent an RTX 3090 24GB on NexGPU at $0.193/GPU-hr, pull the official image, and see it work for pocket change.

How much VRAM does TensorRT-LLM need, and why does a 9GB model still OOM?

VRAM = weights + KV cache + activation peak, and the OOM is usually the second term. `free_gpu_memory_fraction` defaults to 0.9, so the server claims 90% of free VRAM for the KV pool at startup; add the activation cost implied by `--max_num_tokens` and it tips over. Drop it to 0.80–0.85 and lower `--max_num_tokens` and most OOMs vanish immediately. Rough math: FP8 weights ≈ parameter count in GB (8B ≈ 9GB), W4A16 ≈ a little over half (70B ≈ 40GB), everything left goes to KV. To map that curve at your real concurrency, NexGPU rents everything from 24GB to 141GB by the second — measure, then stop the instance.

Can I run TensorRT-LLM on a Tesla V100, Tesla T4 or Tesla P40?

No, and it is not worth the attempt. The current support matrix covers Blackwell, Hopper, Ada Lovelace and Ampere plus GB200 NVL72 and Grace Hopper. Turing (T4) and Volta (V100) are not listed, and Pascal (P40) fell off long before that; NVIDIA states clearly that unlisted architectures are neither developed against nor tested. The good news on NexGPU: the cheapest **supported** card is also the cheapest card we rent — RTX 3090 24GB at $0.193/GPU-hr, below the T4's $0.298, and being Ampere it runs AWQ and GPTQ fine.

Which quantization formats work on RTX 4090 versus RTX 5090, and how do FP8 and NVFP4 differ?

The RTX 4090 is Ada (sm89): real FP8 tensor cores, so FP8 per-tensor and FP8 KV Cache both work, along with W4A16 AWQ and GPTQ — but no NVFP4. The RTX 5090 is consumer Blackwell (sm120), adding NVFP4 and MXFP4 on top of FP8 per-tensor and FP8 KV Cache, which is why natively-MXFP4 weights such as GPT-OSS belong on a 5090 rather than being dequantized elsewhere. Note that the full FP8 block-scaling, rowwise and NVFP4 KV Cache set is datacenter Blackwell (sm100/sm103) only. NexGPU stocks both: 4090 at $0.540/GPU-hr and 5090 at $0.723/GPU-hr — rent one of each and run the same trtllm-bench dataset on both.

TensorRT-LLM or vLLM — which should I pick?

TensorRT LLM's edge is that NVIDIA wrote it: custom kernels for attention, GEMMs and MoE; a complete FP8/NVFP4 quantization path via ModelOpt; and prefill-decode disaggregated serving plus Wide Expert Parallelism, which are what actually separate the pack on very large MoE models. Speculative decoding covers EAGLE-3, MTP, PARD, NGram and more. The price is a pickier environment — CUDA 13.1, PyTorch 2.10, Linux only. vLLM is faster to get going and covers more architectures. The sane answer is to benchmark both on identical traffic. NexGPU's 2,000+ prebuilt images include vLLM and PyTorch, so two per-second-billed instances side by side costs about as much as lunch.

TensorRT-LLM won't install and my CUDA version doesn't match. Now what?

Check three things first. It supports only Linux x86_64 or aarch64 — there is no Windows build, which is the most common cause of death. The current development line wants CUDA 13.1, PyTorch 2.10.0 and Python 3.12, validated on Ubuntu 24.04. And with pip you must pin torch before installing tensorrt_llm, or dependency resolution will swap your torch out from under you. The path of least resistance is not installing at all — use the official nvcr.io release image. NexGPU instances give you SSH, Jupyter, a web terminal and a REST API, so you pull the image and go, and environment problems stop being a category. If you do get stuck, support is bilingual over Telegram with no ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.