Text LLM

Run Olmo 3 on your own GPU: 4.47GB for 7B, 19.48GB for 32B

Ai2's Olmo 3 is one of the very few model families that ships the pretraining data, the post-training recipe and every intermediate checkpoint alongside the weights — Apache 2.0, 65,536-token context. This page gives you the real per-variant VRAM numbers, the single-GPU configurations, and the traps.

OLMo · self-hosted

Olmo 3 comes from the Allen Institute for AI (Ai2), launched November 2025 in two sizes — 7B and 32B — each with Base, Instruct and Think forms. What separates it from other "open" models is not the weights but the full model flow: Dolma 3 for pretraining (roughly 5.9T tokens mixed out of a 9.3T pool, plus the Dolmino mid-training and Longmino long-context stages), Dolci for post-training (SFT, DPO and RLVR sets), and the intermediate checkpoints from every stage, all under Apache 2.0. In December 2025 Olmo 3.1 resumed the best RL run for another 21 days on 224 GPUs for the 32B Think and Instruct models, gaining 5+ points on AIME and 20+ on IFBench — with no architecture change at all.

A few architectural numbers determine what card you need. Both sizes are Olmo3ForCausalLM with max_position_embeddings of 65,536, held up by YaRN (factor 8.0, extrapolated from 8,192), and an interleaved attention pattern: every fourth layer is full attention, the rest use a 4,096-token sliding window. The 7B is 32 layers, hidden 4096, 32 query heads against 32 KV heads — that is MHA, not GQA, which is genuinely unusual for a 7B in 2026 and means a much fatter KV cache than you would expect, mitigated only because 24 of the 32 layers are windowed. The 32B is 64 layers, hidden 5120, 40 query heads against 8 KV heads — ordinary GQA, and therefore the cheaper of the two on KV at long context. Vocabulary is 100,278, rope_theta 500,000.

Deployment is not entirely frictionless, and it is worth knowing that up front. transformers must be 4.57.0 or newer or it will not even recognise the olmo3 model_type. The official config.json ships with use_cache set to false, so a naive generate() call crawls. On the vLLM side there has been an issue where online serving produced gibberish while the offline LLM class worked fine, and an open issue where auto tool choice never emits structured tool calls. SGLang still has no native Olmo3 class registered and falls back to the transformers backend. The GGUF route is the steadier one: llama.cpp has supported it since b7120, and Ollama carries both olmo-3 and olmo-3.1. So the cheapest order of operations is always the same — rent a per-second-billed box, nail the version combination and a smoke test first, then decide how big a card you actually need.

01 —

Which Olmo 3 variants exist, and what each one costs in VRAM

GGUF sizes are the actual file sizes from bartowski's quant repos; bf16 is raw weight size. Context is 65,536 across the board.

Version	Parameters	VRAM	Context	Notes
Olmo 3.1 32B Think	32B (64 layers / 40 query heads / 8 KV heads, GQA)	bf16 ≈64.5GB / Q8_0 34.25GB / Q6_K 26.45GB / Q4_K_M 19.48GB	65,536	The flagship reasoning model and the first fully open 32B thinking model. MATH 96.1, HumanEvalPlus 91.4, IFEval 89.0. It emits <think> tags; Ai2 recommends temperature 0.6, top_p 0.95, max_tokens 32768 — it will happily spend all 32k on a chain of thought, so budget both KV and request timeouts accordingly.
Olmo 3.1 32B Instruct	32B	bf16 ≈64.5GB / Q8_0 34.25GB / Q4_K_M 19.48GB / IQ4_XS 17.33GB	65,536	The non-reasoning chat and function-calling variant — its default system prompt literally describes a function-calling assistant. Pick it for multi-turn dialogue, RAG backends and tool orchestration; time-to-first-token is an order of magnitude better than Think. Note the open vLLM issue about auto tool choice not producing structured tool calls.
Olmo 3 7B Think	7B (32 layers / 32 KV heads, MHA)	bf16 ≈14.6GB / Q4_K_M ≈4.5GB	65,536	Chain-of-thought reasoning on a small card, same <think> tags and same temperature 0.6 / top_p 0.95 / max_tokens 32768 guidance. Good for self-hosted math and code work, but its MHA attention makes the KV cache grow faster with context than a comparable GQA 7B.
Olmo 3 7B Instruct	7B	bf16 14.60GB / Q8_0 7.76GB / Q5_K_M 5.21GB / Q4_K_M 4.47GB / IQ4_XS 4.00GB	65,536	The sweet spot for single-GPU self-hosting: MATH 87.3, IFEval 85.6, HumanEvalPlus 77.2. At Q4_K_M it is 4.47GB, so a 24GB card carries the weights and a full 64K context with room to spare. This is where almost every self-hosted deployment should start.
Olmo-3-1025-7B / Olmo-3-1125-32B (Base)	7B / 32B	bf16 ≈14.6GB / ≈64.5GB	65,536	The base repos carry a date stamp in the name, so searching for "Olmo-3-7B" finds nothing — this trips people up constantly. The 7B base saw 5.93T tokens of dolma3_mix-6T-1025; the 32B base saw 5.50T of dolma3_mix-5.5T-1125. Per-stage intermediate checkpoints (stage1-stepXXX and friends) are all published, so continued pretraining or your own post-training run is on the table.
Olmo 3.1 RL-Zero 7B (Math / Code)	7B	bf16 ≈14.6GB	65,536	RLVR applied directly to the base model, refreshed in Olmo 3.1 with longer and more stable runs. These are clean baselines for RL researchers, not chat assistants — do not benchmark them on dialogue and conclude Olmo is weak.

02 —

Which NexGPU card to rent for Olmo 3

Sized as weights plus KV cache. The 7B is MHA and needs ~9.5GB of KV at 64K; the 32B is GQA and needs only ~4.8GB at the same context — the bigger model is the cheaper one on KV, which is Olmo 3's counterintuitive quirk.

Olmo 3 7B Instruct / Think at Q4_K_M (4.47GB) for smoke tests and batch evals
RTX 3090 24GB$0.193/GPU-hr
The cheapest 24GB card on the network — cheaper than the 16GB Tesla T4 at $0.298 — and a quantised 7B plus a full 64K context still leaves headroom. This is the right card for step one of any Olmo 3 project.
Olmo 3 7B in bf16 with the full 65,536 context (14.6GB weights + ~9.5GB KV)
RTX 5090 32GB$0.723/GPU-hr
With 32 KV heads of MHA, the eight full-attention layers alone consume roughly 8GB of KV at 64K; add the weights and a 24GB card is pinned against the ceiling. 32GB leaves room for concurrency and CUDA graphs.
Olmo 3.1 32B Think at Q4_K_M (19.48GB) serving long chains of thought
RTX A6000 48GB$0.817/GPU-hr
The 32B only needs ~4.8GB of KV at 64K, but Think can emit 32,768 output tokens per request, so 48GB lets you run several concurrent streams without cutting --max-model-len. Single-stream only? RTX 5090 32GB at $0.723/GPU-hr fits it too.
Olmo 3.1 32B in bf16 original weights behind vLLM in production
A100 PCIE 80GB$0.824/GPU-hr
64.5GB of weights plus ~4.8GB of KV at 64K lands around 69GB — one card starts, but you will be tuning --gpu-memory-utilization and trimming max-model-len. For full 64K with real concurrency, take two ($1.648/hr) or move to A100 SXM4 80GB at $1.088/GPU-hr for NVLink.

03 —

Deploying Olmo 3 from scratch

Four steps, all of them inside a NexGPU instance over SSH, Jupyter or the web terminal.

01
Boot a vLLM instance and pin your versions
Pick a vLLM or PyTorch image from the 2,000+ prebuilt catalogue — no quota request, no setup fee. Olmo 3's model_type is olmo3 and anything below transformers 4.57.0 throws an unknown-architecture error. Just as important is the YaRN rope_scaling in the config (factor 8.0, original_max_position_embeddings 8192): an engine that ignores it will silently truncate you to 8,192 tokens.
```
pip install -U "transformers>=4.57.0" vllm && python -c "import transformers; print(transformers.__version__)"
```
02
Pull only the precision you are actually going to run
The 32B in bf16 is ~64.5GB; Q4_K_M is 19.48GB. That is more than a 3x difference. NexGPU storage runs at a $0.414/GB-month median, so over-downloading costs real money — and remember compute billing stops when the instance stops while storage keeps accruing until you destroy it.
```
hf download allenai/Olmo-3.1-32B-Think --local-dir /workspace/olmo31-32b-think
```
03
Serve an OpenAI-compatible endpoint — but smoke-test offline first
One vLLM command gives you /v1/chat/completions. Still, the community has reported online serving producing gibberish while the offline Python LLM class behaved perfectly (vllm-project/vllm#29478 on v0.11.2, eventually closed as not planned), so generate one sentence offline before you flip on the server. On the transformers path, remember to turn use_cache back on — the shipped config has it set to false.
```
vllm serve /workspace/olmo31-32b-think --served-model-name olmo3 --max-model-len 65536 --gpu-memory-utilization 0.92
```
04
Want a result in minutes? Take the GGUF route
llama.cpp has supported Olmo 3 GGUFs since b7120 (the 32B quants were built on b7340), and Ollama carries both the olmo-3 and olmo-3.1 lines — the default olmo-3.1:32b is a 19GB Q4 with a 64K window. The prompt format is <|im_start|> / <|im_end|>, i.e. plain ChatML. On a single RTX 3090 this gets you a first token in minutes.
```
ollama run olmo-3.1:32b-think
```

What an Olmo 3 run actually costs

Metered per second, priced per hour, no minimum, no setup fee, no quota request. A concrete case: you want to reproduce AIME and IFEval on Olmo 3.1 32B Think. Take an RTX A6000 48GB at $0.817/GPU-hr and run Q4_K_M (19.48GB). Half an hour to download weights and start the server, 7.5 hours of evaluation — 8 hours total, so 0.817 × 8 = $6.54. Weights plus logs occupy 25GB, kept three days then destroyed: 25 × $0.414 × 3 ÷ 30 = $1.04. Export 2GB of results: 2 × $0.0081 = $0.02. Total $7.60. Do the same on an A100 PCIE 80GB in bf16 and the compute is comparable at $0.824 × 8 = $6.59, but the weights alone are 64.5GB, so three days of storage costs 64.5 × $0.414 × 3 ÷ 30 = $2.67, landing at $9.28 with egress — quantisation saves you more than VRAM. Cheaper still: debug the pipeline on the 7B first. RTX 3090 24GB at $0.193/GPU-hr for three hours is 0.193 × 3 = $0.58. Stop the instance and compute billing stops that second; storage keeps running until you destroy it.

04 —

FAQ

How much VRAM does it actually take to run Olmo 3 locally?

By tier. 7B: Q4_K_M 4.47GB, Q8_0 7.76GB, bf16 14.60GB. 32B: Q4_K_M 19.48GB, Q8_0 34.25GB, bf16 roughly 64.5GB. Then add KV cache — the 7B is MHA with 32 KV heads and needs about 9.5GB at 64K, while the 32B is GQA with 8 KV heads and needs only about 4.8GB at the same context. So an RTX 3090 24GB at $0.193/GPU-hr is more than enough for a quantised 7B, a quantised 32B starts at RTX 5090 32GB ($0.723/GPU-hr), and bf16 32B wants an A100 PCIE 80GB ($0.824/GPU-hr). All of these spin up and stop in seconds on NexGPU, so being wrong costs you cents.

Can a 24GB RTX 4090 run Olmo 3.1 32B Think?

Q4_K_M weights are 19.48GB, which nominally fits in 24GB — but Think can emit 32,768 output tokens per request, and with KV on top a full 64K window hits the wall. Either cut --max-model-len to around 16K, or drop to IQ4_XS at 17.33GB. If you would rather not fight it, go straight to RTX 5090 32GB ($0.723/GPU-hr) or RTX A6000 48GB ($0.817/GPU-hr); the gap over an RTX 4090 ($0.540/GPU-hr) is not a decision-grade cost per hour. All three are available on NexGPU right now, and you stop paying compute the moment you stop the instance.

How is Olmo 3 different from OLMo 2, and should I still use OLMo 2?

The gap is much wider than the version number suggests. OLMo 2 is Olmo2ForCausalLM with a 4,096-token context, in 7B / 13B / 32B, with the 32B Instruct released in March 2025. Olmo 3 moves to Olmo3ForCausalLM with a 65,536-token context — a full 16x — adds interleaved 4,096 sliding-window attention and YaRN extrapolation, and introduces the Think variants with explicit chains of thought. Unless you are doing a reproduction that must line up with specific OLMo 2 checkpoints, start new projects on Olmo 3.1. Want to compare both lines side by side? Spin up two per-second-billed NexGPU instances, run them in parallel, stop them together.

Is Olmo 3 usable commercially, and what about the licence and data provenance?

Apache 2.0. And it is not just the weights: the Dolma 3 pretraining data, the Dolci post-training data, per-stage intermediate checkpoints, training code and logs are all public, with Ai2's Responsible Use Guidelines worth reading before you ship. That auditability is the real difference from open-weights-but-closed-data models — your compliance team has something to actually inspect. To run it inside an environment you control, NexGPU spans 51 countries and regions with 1,175 verified rentable nodes and 2,498 GPUs, so you can pick where the data lands.

Why is my Olmo 3 emitting gibberish, or only honouring 8K of context?

Three usual suspects. First, transformers below 4.57.0 does not know the olmo3 model_type. Second, your engine does not support the YaRN rope_scaling in the config (factor 8.0, original_max_position_embeddings 8192), so it quietly treats the model as 8,192 tokens and truncates long documents. Third, the shipped config.json has "use_cache": false verbatim, which makes a plain transformers generate() painfully slow. Beyond that, vLLM has had an online-serve gibberish issue (#29478) and still has an open one where auto tool choice never emits structured tool calls (#32534), and SGLang has no native Olmo3 class registered, running it through the transformers backend fallback (sgl-project/sglang#31175). Rent a per-second-billed NexGPU box, find the working version combination once, and you will save more time than reinstalling environments locally — and our bilingual support sits on Telegram with no ticket queue if you get stuck.

What GPU do I need to fine-tune Olmo 3?

For the 7B with QLoRA, a 4-bit base is about 4.5GB and one RTX 4090 24GB ($0.540/GPU-hr) is enough. For bf16 LoRA you need room for 14.6GB of weights plus optimizer state and activations, so an RTX A6000 48GB ($0.817/GPU-hr) is far more comfortable — frameworks like Axolotl already ship Olmo 3 7B LoRA recipes. Full-parameter training on the 32B belongs on multiple A100 SXM4 80GB ($1.088/GPU-hr) or H100 SXM 80GB ($3.582/GPU-hr). Olmo 3's real advantage here is that the base models publish per-stage intermediate checkpoints, so you can resume from mid-training instead of only bolting an adapter onto final weights. NexGPU offers up to 14 GPUs per node and a maximum node VRAM of 2,152GB, with no quota request for multi-GPU jobs.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.