Inference runtime

Self-hosting Ollama: how the VRAM math actually works

Ollama compresses an open-weight LLM stack into one curl command and one port. What stops you is never the command — it's the VRAM.

Ollama · self-hosted

Ollama is an MIT-licensed local LLM runtime; the repo lives at github.com/ollama/ollama and the model library at ollama.com/library. It wraps the llama.cpp quantised-inference stack into three things: an install script, an `ollama run` command, and an HTTP service listening on port 11434. The current stable line is 0.32.x, with 0.33 in the preview channel. Alongside the native /api/chat, /api/generate and /api/embed endpoints it speaks the OpenAI SDK dialect, and `ollama launch` wires a local model straight into Claude Code, VS Code, JetBrains, Zed and n8n.

But Ollama is not a model — it's a shell. VRAM decides what you can actually run, on two levels. First, will it fit: the GB figure on ollama.com/library is the weight file size, not peak VRAM, and you still owe the KV cache another 20–50% on top, scaling with context length. Second, and less obvious: Ollama picks a default context window based on available VRAM — 4K below 24GB, 32K between 24 and 48GB, and 256K only at 48GB and above. So VRAM doesn't just gate whether a model loads, it sets the window you get. For agents, coding assistants and web search, the project's own floor is 64K.

When it doesn't fit, you don't get an error — you get a silent slowdown. Ollama pushes the overflow layers to the CPU, the Processor column in `ollama ps` flips from 100% GPU to a mix, throughput drops by an order of magnitude, and you may not notice until an entire eval run is done. Rather than fighting quantisation levels on a 16GB laptop, rent a card that genuinely fits. NexGPU has 1,175 verified rentable nodes across 51 countries and regions, 2,498 GPUs, 75 GPU models, up to 14 GPUs per node and 2,152GB max node VRAM. Of the 2,000+ prebuilt images, PyTorch, vLLM and Ubuntu CLI are all one click away — SSH in, run one curl, and Ollama is up — compute billing stops the second the instance stops, with no minimum, no setup fee and no quota request.

01 —

The model tiers worth running on Ollama

Sizes are the weight-file sizes published on ollama.com/library; real VRAM use sits above these once KV cache and context are accounted for.

Version	Parameters	VRAM	Context	Notes
gpt-oss:20b	20B (MXFP4, 4.25 bits per parameter)	14GB of weights; the project states it runs on systems with as little as 16GB memory	128K	The best single-card starting point. MXFP4 squeezes the 90%+ of parameters held in MoE weights down to 4.25 bits, and tool calling plus reasoning work out of the box.
gpt-oss:120b	120B MoE (MXFP4)	65GB of weights; explicitly documented as fitting on a single 80GB GPU	128K	The largest frontier-class open model that still loads whole onto one card. No tensor parallelism, no sharding, no multi-GPU debugging — the value inflection point for private inference.
qwen3.5:9b / :27b / :35b	9B / 27B / 35B	6.6GB / 17GB / 24GB	256K	A multimodal family built on Gated Delta Networks plus sparse MoE, taking text and image input. The 9b variant is what the `latest` tag resolves to.
gemma4:12b / :26b / :31b	12B / 25.2B (3.8B active) / 30.7B	7.6GB / 18GB / 20GB	256K	From Google DeepMind, text and image input. The 26b is MoE: 25.2B parameters but only 3.8B active, so it costs 18GB of VRAM while decoding closer to a small model's speed.
deepseek-r1:14b / :32b / :70b	14B / 32B / 70B distilled	9.0GB / 20GB / 43GB	128K	Reasoning distillations that beat same-size general models on maths and code. The full 671B original is a 404GB pull — multi-GPU node territory only.
qwen3.5:122b	122B MoE	81GB of weights — just over the 80GB single-card ceiling	256K	That 81GB figure is genuinely annoying: an 80GB A100 or H100 misses it by a hair. Either move to a 141GB card or accept the overhead of sharding.

02 —

Pick the card for the tier you're running

Metered per second, priced per hour. No minimum, no setup fee, no quota request. List rates below are per GPU-hour.

7B–14B quantised models for prototyping and batch evals (qwen3.5:9b at 6.6GB, deepseek-r1:14b at 9.0GB)
RTX 3090 24GB$0.193/GPU-hr
24GB holds anything under 14GB of weights entirely on-card with room left for a 4K–32K KV cache — currently the best tokens-per-dollar tier we rent.
gpt-oss:20b (14GB), gemma4:31b (20GB) or deepseek-r1:32b (20GB) with a 32K window
RTX 5090 32GB$0.723/GPU-hr
32GB clears Ollama's 24GB threshold to get the 32K default window, leaves a dozen-plus GB for KV cache above a 20GB model, and Blackwell's compute capability 12.0 is on the supported list.
gpt-oss:120b (65GB) on one card, or any agent workload that needs the 256K window
A100 PCIE 80GB$0.824/GPU-hr
gpt-oss:120b is documented as fitting a single 80GB GPU, and 48GB+ is what unlocks the 256K context tier — at $0.824 this is seven cents more than the RTX A6000 48GB at $0.817, so take the 80GB card.
qwen3.5:122b (81GB of weights), or pushing OLLAMA_NUM_PARALLEL past 8 for multi-user serving
H200 141GB$6.660/GPU-hr
81GB of weights overshoots any 80GB card; 141GB swallows it and still leaves roughly 60GB for concurrent KV cache, skipping the interconnect overhead and debugging time of a sharded setup.

03 —

Four steps to Ollama on a rented GPU

From bare instance to an OpenAI-compatible endpoint your apps can call, in under ten minutes.

01
Spin up an instance and install Ollama
Pick the card by VRAM in the console: RTX 3090 24GB for 7B–14B, RTX 5090 32GB for 20B–32B, A100 PCIE 80GB for 120B. Ubuntu CLI or PyTorch images both work. SSH in and run the official installer — it registers the systemd service and detects the driver. Hard requirements: NVIDIA compute capability 5.0+ and driver 550+, or 570+ for compute capability 5.0–6.2 cards.
```
curl -fsSL https://ollama.com/install.sh | sh
```
02
Fix the three defaults that will bite you
Ollama binds to 127.0.0.1:11434 only, so nothing remote can reach it. Context is auto-sized from VRAM, meaning just 4K below 24GB. KV cache defaults to f16. One systemd override fixes all three: bind 0.0.0.0, raise context to 64K, drop KV cache to q8_0 (roughly half the memory of f16; q4_0 is a quarter) and force Flash Attention on. Note that 0.0.0.0 exposes an API with no authentication — pair it with a firewall rule or an SSH tunnel.
```
sudo mkdir -p /etc/systemd/system/ollama.service.d && printf '[Service]\nEnvironment="OLLAMA_HOST=0.0.0.0:11434"\nEnvironment="OLLAMA_CONTEXT_LENGTH=65536"\nEnvironment="OLLAMA_KV_CACHE_TYPE=q8_0"\nEnvironment="OLLAMA_FLASH_ATTENTION=1"\n' | sudo tee /etc/systemd/system/ollama.service.d/override.conf && sudo systemctl daemon-reload && sudo systemctl restart ollama
```
03
Pull a model, then confirm it is actually 100% on the GPU
This is the step people skip and pay for. The Processor column in `ollama ps` tells you whether the model is 100% GPU or split across GPU and CPU. Any CPU percentage means layers spilled into system RAM and throughput has dropped by an order of magnitude — either lower OLLAMA_CONTEXT_LENGTH or move to a bigger card. Logs are in `journalctl -u ollama`.
```
ollama pull gpt-oss:20b && ollama run gpt-oss:20b "Explain MoE in one sentence" && ollama ps
```
04
Wire port 11434 into your own application
Native endpoints are /api/chat, /api/generate and /api/embed, and the OpenAI SDK works if you just point base_url at the instance — no code changes. For multiple users, raise OLLAMA_NUM_PARALLEL (default 1) alongside OLLAMA_MAX_LOADED_MODELS (default 3x the GPU count). Models unload after 5 idle minutes by default; for a long-running service set OLLAMA_KEEP_ALIVE to -1 to pin them in VRAM.
```
curl http://<your-node-ip>:11434/api/chat -d '{"model":"gpt-oss:20b","messages":[{"role":"user","content":"hello"}],"stream":false}'
```

What a week of model selection actually costs

Say you want to run three candidates over one week before committing to production: gpt-oss:20b (14GB), qwen3.5:27b (17GB) and deepseek-r1:32b (20GB) — 51GB of weights combined. Take an A100 PCIE 80GB at $0.824/GPU-hr: all three can stay resident, and 80GB puts Ollama's auto-sized context in the 256K tier. Run four hours a day for five days: compute = 20 hours x $0.824 = $16.48. The 51GB of model files on disk, at the $0.414/GB-month median and kept for only 7 days, = 51 x 0.414 x 7 / 30 = $4.93. Export 2GB of eval logs and egress = 2 x $0.0081 = $0.02. Total for the week: about $21.43. Two things to remember: compute billing stops the second the instance stops, but storage keeps billing until you destroy it. And if you only need to sanity-check 7B–14B quantised models, swap to an RTX 3090 24GB at $0.193/GPU-hr — the same 20 hours costs $3.86.

04 —

FAQ

How much VRAM does Ollama actually need?

Start with the size on ollama.com/library — that is the weight file, not peak VRAM. gpt-oss:20b is 14GB, qwen3.5:27b is 17GB, deepseek-r1:32b is 20GB, gpt-oss:120b is 65GB. Add KV cache on top; a working rule is 1.2–1.5x the weight size, rising with context length. So do not expect a 14GB model to run 128K context on a 16GB card. Instead of estimating, rent an RTX 5090 32GB for one hour at $0.723/GPU-hr on NexGPU and read `ollama ps` — the measurement costs less than a coffee.

My `ollama ps` shows a GPU/CPU split. How do I get it fully on the GPU?

A split means the model did not fit, so the overflow layers landed in system RAM and throughput fell by an order of magnitude. Three options: lower OLLAMA_CONTEXT_LENGTH, set OLLAMA_KV_CACHE_TYPE to q8_0 or q4_0 (about 1/2 and 1/4 the memory of f16), or move to a card with more VRAM. The first two trade window and precision for memory; only the third actually solves it. NexGPU rents 75 GPU models from RTX 3090 24GB at $0.193/GPU-hr up to H200 141GB at $6.660/GPU-hr, all metered per second, so re-running on a bigger card costs almost nothing.

Ollama or vLLM — which should I use?

Ollama wins on getting started: one curl to install, one `ollama run` to load, an OpenAI-compatible endpoint on 11434, and pre-quantised GGUF/MXFP4 weights in the library. It suits solo development, model selection and low-traffic internal services. vLLM wins on throughput: continuous batching and PagedAttention extract far more from the same card under concurrency, which is what a public-facing service needs. The usual path is to select with Ollama and ship with vLLM. NexGPU's 2,000+ prebuilt images include PyTorch, vLLM and Ubuntu CLI, so both stacks live on the same box and switching frameworks never means rebuilding the environment.

Is setting OLLAMA_HOST to 0.0.0.0 safe?

No. Ollama's HTTP API has no authentication by default, so the moment 11434 faces the public internet anyone can pull models, run inference and burn your GPU hours. The safe pattern is to keep the default 127.0.0.1 bind and forward 11434 over SSH, or bind 0.0.0.0 but allow only your own IP at the firewall with Nginx auth in front. Every NexGPU instance offers SSH, Jupyter, a web terminal, a REST API and a CLI — the SSH tunnel is the least effort of the five.

Which GPUs does Ollama support? Can old cards like the Tesla P40 or V100 still run it?

Yes. Ollama requires NVIDIA compute capability 5.0+ with driver 550+, and 570+ for compute capability 5.0 through 6.2. The Tesla P40 is Pascal 6.1 and the V100 is Volta 7.0, so both are supported; AMD needs ROCm v7, and Intel and other vendors go through the Vulkan backend. The limitation on old cards is not capability but the lack of native acceleration for newer formats like FP8 and MXFP4 — GGUF Q4 still runs very economically. NexGPU's Tesla V100 32GB is $0.188/GPU-hr and the Tesla P40 24GB is $0.214/GPU-hr, the two cheapest ways to serve a 7B–14B quantised model.

Will pushing context to 128K blow up my VRAM?

Yes, and faster than most people expect. Ollama picks the default window from available VRAM: 4K below 24GB, 32K from 24 to 48GB, and 256K only at 48GB and above — while the project's own floor for agents, coding assistants and web search is 64K. Without changing cards you can set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, but the direct fix is 48GB or more. On NexGPU the RTX A6000 48GB is $0.817/GPU-hr and the A100 PCIE 80GB is $0.824/GPU-hr — seven cents apart, so long-context workloads should just take the 80GB card.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.