Skip to main content
NexGPU

Inference runtime

Self-hosting llama.cpp: one 24GB card, from quantization to a served endpoint

GGUF quantization takes an 8B model down to 4.58GB and a 27B down to 19GB. That leaves two questions: which card, and what does it cost per hour.

Sign up freeView pricing

llama.cpp is the C/C++ inference engine maintained by ggml-org under the MIT license, copyright "The ggml authors". No Python dependency — the build output is a set of binaries you can drop onto bare metal. Builds are tagged bXXXX and ship several times a day: -sm tensor support for DeepSeek V4, GLM-4.5-Air fixes, flattening Mamba2's projections into a single GEMM dispatch all landed within days. That cadence is the whole reason people reach for it instead of waiting on a vendor runtime.

The entry point is not what it was in 2024, though. The official quick start is now a unified llama command: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF pulls a model and starts chatting, llama serve -hf ... brings up an OpenAI-compatible server, and plain text completion has been split out into its own llama-completion binary. Both -ngl and -fa now default to auto, so you no longer count layers by hand. Any tutorial that still tells you to run ./main or pass -no-cnv is going to fail on you.

What actually decides which card you rent is not the parameter count — it's the GGUF file size plus the KV cache. Llama-3.1-8B's f16 KV cache costs roughly 128KB per token (8 KV heads × 128 dims × K and V × 2 bytes × 32 layers), so a 32K context alone eats 4GB. Add -np parallel slots, an mmproj vision projector and MTP draft weights, and 24GB gets tight faster than you'd expect. NexGPU has 1,175 verified rentable nodes, 2,498 GPUs and 75 GPU models, from a Tesla V100 32GB at $0.188/GPU-hour up to a 141GB H200 — all metered per second, so measure first and commit later.

01 —

GGUF quant tiers: measured bits-per-weight and file size

Bits-per-weight and the 8B column come from the measurement table in llama.cpp's tools/quantize (baseline model meta-llama/Llama-3.1-8B). The 27B and 12B figures are the actual file sizes published by ggml-org on Hugging Face.

VersionParametersVRAMContextNotes
Q4_K_M4.8944 bpw8B 4.58GB | 27B 19GB8B with q8_0 KV: 128K fits on a 24GB cardThe default choice. Nearly every GGUF release page lists it first — this is where the quality-versus-size knee sits.
IQ4_XS4.4597 bpw8B 4.17GBLeaves ~0.4GB more for KV than Q4_K_MThe i-quant you reach for when you're right at the edge of VRAM. Recovers accuracy via an imatrix importance matrix; decode overhead is slightly above K-quants.
Q5_K_M5.7036 bpw8B 5.33GB8B with f16 KV: roughly 96K on a 24GB cardThe middle tier for when you suspect Q4 is costing you quality but don't want to pay Q8's footprint. Common A/B reference.
Q6_K6.5633 bpw8B 6.14GBFull 128K for an 8B on a 32GB RTX 5090Quality already hugs Q8 at 1.8GB less. The sweet spot for long-context 8B work on a single 32GB card.
Q8_08.5008 bpw8B 7.95GB | 27B 28.6GB | 30B-A3B 33.6GB27B still leaves ~19GB for KV on a 48GB cardThe near-lossless baseline. Benchmark quantization loss against this, not against F16 — F16's footprint doubles the cost of the comparison run.
F16 / BF1616.0005 bpw8B 14.96GB | 12B 23.8GB | 27B 53.8GB12B BF16 already fills 24GB — start at 48GBThe source for quantization and imatrix calibration. gemma-4-12B-it in BF16 is 23.8GB: the weights technically fit a 24GB card, the KV cache does not. Don't fight it.

02 —

Matching quant tier to card: real NexGPU rates

Size the GGUF plus the KV cache first, then pick the card. Every row below leaves headroom for parallel slots and compute buffers.

  • Smoke-testing an 8B–12B Q4_K_M and wiring up the API

    RTX 3090 24GB$0.193/GPU-hour

    sm_86 is among the most thoroughly exercised architectures in the CUDA backend, and a fully offloaded 4.58GB of weights still leaves a dozen-plus GB for KV. Cheapest way onto 24GB.

  • 27B Q4_K_M with an mmproj vision projector at long context

    RTX A6000 48GB$0.817/GPU-hour

    19GB of weights plus a 931MB mmproj fit on one card, leaving twenty-odd GB for KV and -np slots. No multi-GPU splitting to debug.

  • Q8_0 27B or a 30B-A3B MoE for quality comparison and load testing

    A100 PCIE 80GB$0.824/GPU-hour

    28.6GB / 33.6GB of Q8_0 weights still leave 45GB+ free, and sm_80's Flash Attention kernels are mature enough that llama-bench numbers reproduce.

  • BF16 models across GPUs with tensor parallelism (-sm tensor + NCCL)

    H100 SXM 80GB$3.582/GPU-hour

    The docs state plainly that -sm tensor wants a fast interconnect — only SXM NVLink really holds up. NexGPU nodes take up to 14 GPUs, with a 2,152GB max node VRAM.

03 —

From bare instance to OpenAI-compatible endpoint in four steps

Spin up an NVIDIA instance on NexGPU, get in over SSH, Jupyter or the web terminal, and copy these verbatim.

  1. 01

    Install llama.cpp — the official CUDA image is the shortest path

    ghcr.io carries full, light and server variants, each in CUDA 12 and CUDA 13 flavours. Note that the RTX 5090 is Blackwell (sm_120) and needs the CUDA 13 line, or model load dies with "no kernel image is available for execution".

    docker run --gpus all -p 8080:8080 -v $PWD/models:/models ghcr.io/ggml-org/llama.cpp:server-cuda13 -m /models/model.gguf --host 0.0.0.0
  2. 02

    Pin compute capabilities explicitly when building from source

    GGML_NATIVE tunes for whatever machine you're on, so the binary may not run elsewhere. Turn it off and list every architecture you care about: 86 for 3090 / A10 / A6000, 89 for the 4090, 80 for A100, 90 for H100 and H200, 120 for the 5090, 70 for V100, 61 for the P40.

    cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF -DCMAKE_CUDA_ARCHITECTURES="61;70;80;86;89;90;120" && cmake --build build --config Release -j
  3. 03

    Let llama-fit-params do the VRAM math

    Stop guessing at -ngl. It reads available device memory and prints a usable argument string straight to stdout: a reduced -c, a computed -ngl, and -ot rules naming which layers' FFN tensors get pushed back to CPU. Pipe it into llama-server and go.

    ./build/bin/llama-fit-params --model /models/model.gguf | tee args.txt && cat args.txt | xargs ./build/bin/llama-server --model /models/model.gguf
  4. 04

    Serve it and expose it safely

    llama-server exposes /v1/chat/completions, /v1/completions, /v1/embeddings and an Anthropic-style /v1/messages, plus a built-in web UI. -np opens parallel slots for continuous batching, --jinja uses the model's own template, and moving the KV cache to q8_0 halves that footprint. Always add --api-key before exposing it. Start it with no -m at all and it enters router mode, loading models on demand by the request's model field, with --models-max defaulting to 4 resident instances.

    llama serve -hf ggml-org/gemma-4-12B-it-GGUF:Q8_0 --host 0.0.0.0 --port 8080 -c 32768 -np 4 --jinja -ctk q8_0 -ctv q8_0 --api-key $LLAMA_KEY

What a full quantization run actually costs

Take Llama-3.1-8B end to end: about 15 minutes to pull 14.96GB of F16 weights, ~40 minutes for llama-imatrix to build the importance matrix, ~15 minutes for llama-quantize to emit Q4_K_M (4.58GB), Q5_K_M (5.33GB) and IQ4_XS (4.17GB), and ~50 minutes for llama-perplexity to score all three. Call it 2 hours. On an RTX 4090 24GB that's 2 × $0.540 = $1.08; if you're not in a hurry, an RTX 3090 24GB makes it 2 × $0.193 = $0.386. Pulling the 4.58GB Q4_K_M back home costs 4.58 × $0.0081 ≈ $0.04 at the median egress rate. The line item that actually bites is storage: parking 19GB of 27B weights for a month is 19 × $0.414 ≈ $7.87 — more than the compute. Compute billing stops the second the instance stops; storage keeps running until you destroy the volume. So push your quantized artifacts back to Hugging Face and re-pull with -hf next time rather than paying rent on a disk. Per-second metering, no minimum, no setup fee, no quota request — that's the whole barrier to entry for one experiment.

04 —

FAQ

How much VRAM does llama.cpp actually need, and how large a model fits on a 24GB card?

Size it by GGUF file size plus KV cache, never by parameter count. The upstream measurement table puts Llama-3.1-8B at 4.58GB in Q4_K_M, 7.95GB in Q8_0 and 14.96GB in F16; at the 27B tier, Qwen3.8-27B ships at 19GB in Q4_K_M, 28.6GB in Q8_0 and 53.8GB in BF16. KV is separate: roughly 128KB per token for an 8B at f16, so 32K context is 4GB. Practically: a 24GB card runs a 14B Q4_K_M comfortably, squeezes a 27B Q4_K_M only with the KV cache moved to q8_0, and you want 48GB for 27B at long context. On NexGPU the RTX 3090 24GB is $0.193/GPU-hour and the RTX A6000 48GB is $0.817/GPU-hour, metered per second — rent one for an hour, measure your own model, then decide.

llama.cpp vs vLLM vs Ollama — when should I pick llama.cpp?

llama.cpp wins on quantization density and hardware reach: 1.5-bit through 8-bit integer quantization, CPU+GPU hybrid inference, and a dozen-plus backends spanning CUDA, HIP, Metal, Vulkan, WebGPU, SYCL and CANN, all shipping as dependency-free binaries. vLLM wins on throughput under heavy concurrency. So: single card, tight VRAM, a model larger than the GPU, or unusual hardware — llama.cpp. Dozens of concurrent streams saturating an A100 or H100 — vLLM. Ollama is a wrapper around llama.cpp, so the moment you need -ot, --n-cpu-moe or -sm tensor you're back to llama.cpp directly. NexGPU runs both: vLLM has a prebuilt image among the 2,000+ on offer, and llama.cpp pulls straight from the official ghcr registry — test them side by side on one box.

Why do the ./main and llama-cli -p commands in older tutorials no longer work?

The CLI surface was restructured. The official quick start is now a unified llama command — llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF pulls from Hugging Face and starts a chat, llama serve -hf ... brings up the OpenAI-compatible server — while the old ./main / -no-cnv style plain completion moved into a separate llama-completion binary. Both -ngl and -fa now default to auto, so no more counting layers. The llama-cli, llama-server and llama-completion binaries all still exist; their division of labour changed. Installation options have widened too: winget install llama.cpp, brew install llama.cpp, conda-forge and nix all work. Spin up a per-second-metered NexGPU instance and you can verify old and new invocations in half an hour.

llama.cpp on an RTX 5090 fails with "no kernel image is available for execution" — how do I fix it?

The 5090 is Blackwell, compute capability sm_120, and older prebuilt binaries simply never compiled that architecture in. Two fixes: use the official Docker CUDA 13 variant (swap :server-cuda for :server-cuda13), or build with -DCMAKE_CUDA_ARCHITECTURES="120" explicitly. The same class of failure shows up on older cards — the Tesla P40 is sm_61 and the V100 is sm_70, and the P40 additionally has no fast FP16 path, so it leans on llama.cpp's MMQ integer kernels to be usable at all. Easiest fix is -DGGML_NATIVE=OFF plus a full architecture list in one build. NexGPU has the RTX 5090 32GB at $0.723/GPU-hour, Tesla V100 32GB at $0.188 and Tesla P40 24GB at $0.214, so you can reproduce every one of these architecture quirks on real silicon instead of buying cards to find out.

I'm just barely out of VRAM — what are llama.cpp's options for saving memory?

In increasing order of cost: drop KV cache precision first, since -ctk q8_0 -ctv q8_0 nearly halves it (q4_0, q5_0, iq4_nl and bf16 are also accepted; f16 is the default). Then step down a quant tier — IQ4_XS saves roughly 0.4GB per 8B over Q4_K_M. For MoE models, --n-cpu-moe N keeps the expert weights of the first N layers on CPU, which moves the bulk of the parameters off the GPU in one flag. For finer control, -ot takes a tensor-name regex and a target buffer type. And if you're short by a sliver, GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 on Linux spills into system RAM at a speed penalty. Or skip the trial and error: llama-fit-params computes all of this and prints it. Failing that, just take a bigger card on NexGPU — an A100 PCIE 80GB is $0.824/GPU-hour, and the tuning time you save is worth more than the difference.

How do I run llama.cpp across multiple GPUs, and should I use -sm layer or -sm tensor?

-sm layer is the default: pipeline parallelism, each GPU holding a contiguous slice of layers with the KV cache on the owning device. It tolerates slow interconnects, is the right choice when memory is the constraint or prefill speed matters, and -ts 3,1 lets you hand-tune the split. -sm tensor is the newer tensor-parallel mode that shards both weights and KV; it decodes faster on large dense models but the docs are explicit that it needs a fast interconnect, benefits from a separately installed NCCL, and can be paired with GGML_CUDA_P2P=1 for direct peer transfers (some BIOS/IOMMU combinations crash — turn it off if yours does). The old -sm row is deprecated in favour of tensor. Testing tensor parallelism properly means NVLink: NexGPU's H100 SXM 80GB is $3.582/GPU-hour, nodes take up to 14 GPUs with 2,152GB max node VRAM, and two hours is enough to benchmark both modes and shut it down.

More in Local runtimes

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.