Open multimodal LLM

Self-host Gemma 4 — 31B fits on one 4090

Google DeepMind shipped the whole Gemma 4 line under Apache 2.0: the 31B dense model is 17.5GB at Q4_0, and 26B-A4B activates only 3.8B parameters per forward pass. Here are the real VRAM numbers, the real gotchas, and which card to rent.

Gemma · self-hosted

Gemma 4 launched in April 2026, with the 12B added that June. Five sizes ship today: E2B (5.1B total, 2.3B effective), E4B (8B total, 4.5B effective), 12B Unified (11.95B), 26B-A4B (25.2B total, 3.8B active MoE) and 31B Dense (30.7B). E2B and E4B carry a 128K context; 12B, 26B-A4B and 31B all go to 256K. Over 140 languages, training data cut off January 2025. The biggest change isn't a parameter count though — this is the first Gemma release under Apache 2.0. The old Gemma Terms of Use, with its usage restrictions, is gone, and the weights can go straight into a commercial product.

Three architectural facts matter when you deploy. First, the 12B is encoder-free: image patches and audio waveforms are projected straight into the LLM embedding space by lightweight linear layers instead of passing through separate vision or audio encoders, so text, images, audio and video all flow into one decoder-only transformer — which is how it fits on a 16GB laptop. Second, 26B-A4B is a mixture of experts: all 25.2B parameters sit in VRAM, but only 3.8B fire per token, giving you 4B-class latency at close to 31B quality. Third, Google also published the Gemma 4 Assistant models (for example google/gemma-4-31B-it-assistant) — small text-only draft models that do speculative decoding via Multi-Token Prediction and share the target model's KV cache outright, letting the draft model skip prefill entirely.

On the scoreboard, 31B-it posts 85.2% MMLU Pro, 89.2% AIME 2026, 80.0% LiveCodeBench v6, 84.3% GPQA Diamond and 76.9% on tau2-bench, ranking #3 among open models on the Arena text leaderboard; 26B-A4B lands at 82.6 / 88.3 / 77.1 / 82.3 / 68.2 for the #6 spot. Running a model at that level on your own hardware comes down to one constraint: VRAM. And VRAM you can rent by the second. NexGPU has 1,175 verified rentable nodes across 51 countries and regions, 2,498 GPUs and 75 GPU models, from a $0.188/GPU-hr Tesla V100 up to a 141GB H200.

01 —

VRAM by Gemma 4 variant

Figures from Google's official inference memory table — weights only, KV cache is extra

Version	Parameters	VRAM	Context	Notes
Gemma 4 E2B (google/gemma-4-E2B-it)	5.1B total / 2.3B effective	BF16 11.4GB / SFP8 5.7GB / Q4_0 2.9GB (mobile format 1.1GB)	128K	The smallest edge model — text, image and audio input. The mobile QAT format quantizes token-generation layers to 2-bit while keeping reasoning layers at higher precision, squeezing it to 1.1GB for phones. Good for offline voice assistants and summarization that can't leave the device.
Gemma 4 E4B (google/gemma-4-E4B-it)	8B total / 4.5B effective	BF16 17.9GB / SFP8 8.9GB / Q4_0 4.5GB (mobile format 2.5GB)	128K	The quality inflection point at the edge: GPQA Diamond jumps from E2B's 43.4% to 58.6%. The official vLLM recipe says a single 24GB card is enough, and Q8_0 is the recommended quant here — 4-bit hurts more at 4B scale.
Gemma 4 12B Unified (google/gemma-4-12B-it)	11.95B dense	BF16 26.7GB / SFP8 13.4GB / Q4_0 6.7GB	256K	Encoder-free unified multimodal, and the first mid-size Gemma with native audio in. Text, images, audio and video enter the same decoder. Audio caps at 30 seconds per clip, video at 60 seconds at 1fps. Benchmarks close to 26B-A4B at under half the total memory — the best value if you're standing up a multimodal service.
Gemma 4 26B-A4B MoE (google/gemma-4-26B-A4B-it)	25.2B total / 3.8B active	BF16 57.7GB / SFP8 28.8GB / Q4_0 14.4GB	256K	The concurrency pick. There is deliberately no W4A16 checkpoint — quality loss at 4-bit is too severe for the expert weights — so the vLLM recipe recommends INT8 for roughly 47% memory savings. On the GGUF path, a naive Q4_0 conversion hits a scale mismatch; community dynamic quants (UD-Q4_K_XL) recover top-1 from 70.2% back to 85.6%.
Gemma 4 31B Dense (google/gemma-4-31B-it)	30.7B dense	BF16 69.9GB / SFP8 34.9GB / Q4_0 17.5GB	256K	The flagship. 60 layers with hybrid sliding-window and global attention, a 262K vocabulary, a ~550M-parameter vision encoder, and a configurable image token budget of 70–1120. Text and image input, no audio. That 17.5GB Q4_0 figure is what lets it land on a 24GB consumer card.

02 —

Pick a card by workload — real NexGPU rates

Weights are the floor; KV cache is the variable. Every row below leaves context headroom.

Validate 12B or E4B at Q4_0 before committing
RTX 3090 24GB$0.193/GPU-hr
The 12B Q4_0 weights are just 6.7GB, leaving well over 15GB of the card purely for KV cache — enough for a genuinely long context. The cheapest way to find out whether Gemma 4 solves your problem.
Run 31B at Q4_0 on one card for an internal assistant
RTX 4090 24GB$0.540/GPU-hr
17.5GB of weights in 24GB leaves about 6GB, which holds steady at --max-model-len 32768 without OOM. Ada's int4 kernels are noticeably faster than Ampere's for single-user latency.
Serve 26B-A4B in INT8 under real concurrency
RTX A6000 48GB$0.817/GPU-hr
28.8GB of weights on a 48GB card leaves nearly 20GB for KV cache, matching the INT8 path Google's own vLLM recipe recommends. Only 3.8B parameters activate per token, so throughput far exceeds a dense model of the same footprint.
Production 31B in BF16 with tensor parallelism
A100 SXM4 80GB ×2$1.088/GPU-hr
Two 80GB cards is exactly what the official vLLM recipe specifies for 31B BF16. Force 69.9GB onto one card and there is no KV cache left; TP=2 is what lets you push --max-model-len past 32768 and still leave the continuous-batching scheduler room to admit new requests.

03 —

Getting Gemma 4 running on NexGPU

Four steps from boot to an OpenAI-compatible endpoint

01
Boot an instance with vLLM
Pick a card from the table above at console.nexgpu.net and launch a prebuilt vLLM or PyTorch image — there are 2,000+ to choose from, reachable over SSH, Jupyter or the web terminal. Gemma 4 kernels are still moving fast in vLLM, so install the nightly rather than the stable release; the July 2026 wave brought FA4 on Hopper, chat template fixes and tool-calling patches.
```
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 --index-strategy unsafe-best-match
```
02
Pull the weights
Apache 2.0 means no access request, no gated agreement, no waiting. If you're taking the GGUF route, grab a community dynamic quant rather than a raw Q4_0 conversion — the 26B-A4B scale mismatch is real. Variants with audio need the vllm[audio] extra installed as well.
```
hf download google/gemma-4-26B-A4B-it --local-dir /workspace/gemma-4-26B-A4B-it
```
03
Serve it, with the reasoning and tool parsers attached
Gemma 4's thought blocks and function calls both need dedicated parsers. Without --reasoning-parser gemma4 and --tool-call-parser gemma4 you'll just see raw special tokens leaking into the response body. Set --max-model-len to the longest request you will actually accept, not the 256K the model theoretically supports: KV cache scales linearly with context, and once it's full the continuous-batching scheduler has no room to admit new requests, so the server either drops them or OOMs.
```
vllm serve google/gemma-4-26B-A4B-it --max-model-len 32768 --gpu-memory-utilization 0.90 --reasoning-parser gemma4 --tool-call-parser gemma4 --enable-auto-tool-choice --limit-mm-per-prompt image=4 --host 0.0.0.0 --port 8000
```
04
Verify, and fix your sampling parameters
Google's defaults are temperature 1.0, top_p 0.95, top_k 64 — do not carry over the 0.7 you use for other models. To turn thinking on, prepend <|think|> to the system prompt; to turn it off on the llama.cpp side, pass --chat-template-kwargs '{"enable_thinking":false}'. In multi-turn chats keep only the final visible answer in history and strip prior thought blocks, or the model starts following its own discarded drafts.
```
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"google/gemma-4-26B-A4B-it","messages":[{"role":"user","content":"Explain in one sentence why MoE saves compute"}],"temperature":1.0,"top_p":0.95,"top_k":64}'
```

What a first month actually costs

Say you're turning Gemma 4 into an internal service. Step one: evaluate 31B at Q4_0 on an RTX 4090 24GB for three hours — 0.540 × 3 = $1.62. Step two: stay on that same 4090 and fine-tune 31B with QLoRA (4-bit training measures around 22GB, which just fits in 24GB) for eight hours — 0.540 × 8 = $4.32. Step three: go live, running 26B-A4B in INT8 on an RTX A6000 48GB for a full month — 0.817 × 24 × 30 = $588.24. Add 60GB of storage for weights and checkpoints at 0.414 × 60 = $24.84, plus 200GB of egress at 0.0081 × 200 = $1.62. First month total: 1.62 + 4.32 + 588.24 + 24.84 + 1.62 = $620.64. Two things worth knowing: billing is metered per second, so if that three-hour evaluation finishes in 2h47m you pay 2h47m. And compute billing stops the moment the instance stops, while storage keeps billing until you destroy the volume — so during development, shut down when you step away and leave the weights on disk for next time. No minimum, no setup fee, no quota request.

04 —

FAQ

How much VRAM does Gemma 4 actually need?

It depends on variant and precision. In BF16: E2B 11.4GB, E4B 17.9GB, 12B 26.7GB, 26B-A4B 57.7GB, 31B 69.9GB. SFP8 roughly halves those, and Q4_0 halves them again — 31B drops to 17.5GB, 26B-A4B to 14.4GB, 12B to 6.7GB. All of that is weights only; KV cache is separate and scales with your --max-model-len and concurrency. If you'd rather not do the math, use the table above: an RTX 3090 at $0.193/GPU-hr for quantized evaluation, an RTX 4090 at $0.540 for 31B Q4_0, an RTX A6000 48GB at $0.817 for 26B-A4B in INT8 — all bootable from a prebuilt image on NexGPU.

I already run Gemma 3. Is upgrading to Gemma 4 worth it?

Yes, and not only for the scores. The jump is steep — AIME 2026 went from 20.8% to 89.2%, LiveCodeBench from 29.1% to 80.0%, GPQA from 42.4% to 84.3% — but the harder reason is licensing. Gemma 3 shipped under Google's own Gemma Terms of Use with usage restrictions; Gemma 4 is Apache 2.0 across every size, which is a much shorter conversation with legal. Gemma 4 also lifts context from 128K to 256K on the mid and large models and adds two new architectural paths, MoE and encoder-free multimodality. If you want both generations side by side on one box for a fair comparison, NexGPU supports up to 14 GPUs per node and a maximum node VRAM of 2,152GB.

Can I run 26B-A4B in 4-bit the way I run 31B?

Not the same way. Google deliberately did not publish a W4A16 checkpoint for 26B-A4B because quality loss at 4-bit is too severe — MoE expert weights are more quantization-sensitive than dense ones. The official vLLM recipe calls for INT8 instead, saving roughly 47% of memory and landing at 28.8GB. If you insist on GGUF, avoid a raw Q4_0 conversion (scale mismatch) and take a community dynamic quant, which measurably recovers top-1 accuracy from 70.2% to 85.6%. That 28.8GB figure maps cleanly onto NexGPU's RTX A6000 48GB at $0.817/GPU-hr, leaving nearly 20GB for KV cache.

I enabled thinking, so why is reasoning_content empty in the OpenAI response?

This is a known Gemma 4 rough edge on vLLM: text decoding strips the special tokens before the reasoning parser ever sees them, so Gemma4ReasoningParser never gets the thought block and reasoning_content stays empty. First confirm you passed --reasoning-parser gemma4 and are using the official tool_chat_template_gemma4.jinja. If it's still empty, move to a vLLM nightly — the July wave specifically patched chat templates and tool calling. On the llama.cpp side, use llama-server rather than llama-cli if you need thinking reliably disabled. Spinning up a second NexGPU instance on a different image to test takes under a minute, so the cost of checking is close to nothing.

Can I use Gemma 4 in a commercial product?

Yes. Gemma 4 is the first Gemma release under Apache 2.0, covering all five sizes and every format including the QAT quantized weights. Commercial use, redistribution and closed-source packaging are all within the licence, with nothing to request from Google. It also means your fine-tuned weights are genuinely your asset — park them on NexGPU storage at a $0.414/GB-month median rate, keep the instance stopped, and you pay nothing for compute until you need the model again.

Can a 24GB card fine-tune 31B?

Yes, with QLoRA. 4-bit training on 31B measures around 22GB, which fits inside 24GB. For comparison, E2B LoRA needs 8–10GB, E4B around 17GB, and 26B-A4B needs 40GB+ because every expert weight has to be resident. So NexGPU's RTX 4090 24GB at $0.540/GPU-hr handles 31B fine-tuning, while 26B-A4B wants an RTX A6000 48GB at $0.817/GPU-hr or an A100 PCIE 80GB at $0.824/GPU-hr. Per-second billing means a failed run costs you minutes, not hours — and if something breaks, our bilingual support is on Telegram with no ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.