Vision-language model

Self-hosting PaliGemma: the 3B weighs just 6.1GB — one 24GB card runs detection, OCR and segmentation

PaliGemma 2 is a transfer-oriented VLM built from a SigLIP-So400m vision tower and a Gemma 2 language tower. bf16 weights are 6.1GB, 19.3GB and 55.3GB for the 3B, 10B and 28B. The cheapest NexGPU card that saturates the 3B is an RTX 3090 24GB at $0.193/GPU-hour.

PaliGemma · self-hosted

PaliGemma is not a chatbot, and Google says so on the model card: it is not a multi-turn conversational model, it is designed for a single round of image-plus-text input. Input is a fixed square image (224, 448 or 896) plus a prefix; output is a caption, an answer, bounding-box coordinates, or segmentation codewords. The whole family loads through one class, PaliGemmaForConditionalGeneration, and the safetensors parameter counts are exactly 3,033,127,152 (3B), 9,663,523,568 (10B) and 27,651,371,760 (28B), with Gemma 2 2B / 9B / 27B as the language towers.

PaliGemma 2 is the current line — there is no PaliGemma 3. General multimodal capability moved into mainline Gemma: Gemma 3 onward is natively multimodal, and Gemma 4 handles variable aspect ratios plus video and audio across every size. So if you want an assistant that converses, reads long documents and reasons over multiple images, PaliGemma is the wrong tool. What keeps it downloaded is a different job: it is a base model built for transfer. The vocabulary reserves 1024 <loc0000>–<loc1023> position tokens and 128 <seg000>–<seg127> segmentation codewords, tasks are switched with a plain text prefix (cap, caption, describe, ocr, answer, question, detect, segment), and fine-tuning it into a model that does exactly one visual job well is cheap. The most-downloaded PaliGemma 2 checkpoint on the Hub is paligemma2-3b-ft-docci-448 — a task-specific long-captioning build. That is how people actually use it.

Three things to know before you self-host. One: the Hub repos are gated — you must accept the Gemma terms on the model page and authenticate with a token, so a bare wget in CI gets a 403. Two: official weights are bf16 only, and neither the Turing-era Tesla T4 nor the Volta-era V100 has native bf16, while Gemma-family activations tend to overflow in fp16 — do not run native precision on those cards. Three: the GGUF path is essentially empty. The Hub has only a handful of community conversions of the original PaliGemma with three-digit download counts. The quantisation routes that actually work are bitsandbytes NF4, torchao int4 (the transformers docs demo int4_weight_only on the 28B itself), and MLX 4-bit builds on Mac. For production serving, vLLM lists PaliGemmaForConditionalGeneration as supported, with LoRA and pipeline parallelism both ticked.

01 —

Every PaliGemma checkpoint and what it costs in VRAM

mix works out of the box, pt is only a base, ft is a finished product — pick the wrong suffix and you get gibberish.

Version	Parameters	VRAM	Context	Notes
google/paligemma2-3b-mix-448	3.03B (SigLIP-So400m + Gemma 2 2B)	bf16 weights ~6.1GB, budget 10GB on the card; ~1.6GB quantised to int4	448×448 → 1024 image tokens	The default starting point. Mix is multi-task tuned, so caption / ocr / detect / segment prefixes all work immediately, and a 24GB card still has room for a large batch pipeline.
google/paligemma2-10b-mix-448	9.66B (Gemma 2 9B language tower)	bf16 weights ~19.3GB, ~21–22GB with KV cache and activations; ~5GB at int4	448×448 → 1024 image tokens	The next tier up when the 3B struggles on charts, tables and dense small type. 19.3GB is uncomfortably tight on a 24GB card; 32GB is the sane floor.
google/paligemma2-28b-mix-448	27.65B (Gemma 2 27B language tower)	bf16 weights ~55.3GB — needs an 80GB card; ~15GB after torchao int4	448×448 → 1024 image tokens	The ceiling of the family, downloaded roughly a hundredth as often as the 3B. Most people never need it, but it is the exact model the official transformers int4 quantisation example uses.
google/paligemma2-3b-pt-896 (pt series, at 224 / 448 / 896)	pt exists at all three sizes: 3.03B / 9.66B / 27.65B	3B-896 weights are still ~6.1GB, but 4096 image tokens drive prefill memory and compute up sharply	896×896 → 4096 image tokens	Pretrained base only, meant for fine-tuning. Prompt it directly and you get meaningless output — that is not a broken download. 896 is a price worth paying only for dense recognition: invoices, formulas, music scores, molecular structures.
google/paligemma2-3b-ft-docci-448 / paligemma2-10b-ft-docci-448	3.03B / 9.66B	bf16 ~6.1GB / ~19.3GB	448×448 → 1024 image tokens	Fine-tuned on DOCCI for long, fine-grained descriptions, and the most-downloaded PaliGemma 2 checkpoint on the Hub. If you want one image turned into an accurate paragraph, use this instead of training from pt.
google/paligemma-3b-mix-448 (the original PaliGemma)	2.92B (Gemma 1 2B language tower)	the main branch stores fp32, so the download is ~11.7GB; pass revision='bfloat16' and it drops to ~5.8GB	448×448 → 1024 image tokens; an 896 build exists too	Gen 1 only ever came in 3B. Unless you are reproducing older results or need its ft-docvqa / ft-refcoco-seg checkpoints, start new projects on PaliGemma 2.

02 —

Pick the card by scenario: what PaliGemma actually needs

Weight size is the measured safetensors parameter count × 2 bytes; everything else goes to image tokens, KV cache and activations.

Batch inference with 3B mix / ft-docci — captioning, OCR and detection labels over tens or hundreds of thousands of images
RTX 3090 24GB$0.193/GPU-hour
6.1GB of weights leaves 17GB+ for a fully saturated batch pipeline, and Ampere has native bf16 so you never risk Gemma overflowing in fp16.
10B mix-448 single-card bf16 inference, or the 3B at 896 for dense documents
RTX 5090 32GB$0.723/GPU-hour
19.3GB of weights leaves too little headroom on 24GB for a 4096-image-token full-attention prefill; 32GB is the cheapest safe line for this tier.
LoRA fine-tuning the 3B or 10B into a detection or OCR head for your own domain
RTX A6000 48GB$0.817/GPU-hour
LoRA only trains adapters, so memory goes to activations over long 448/896 image sequences — 48GB fits a bigger batch and the peak you hit with gradient checkpointing off.
28B mix-448 at native bf16, or a full-parameter fine-tune of the 3B with AdamW optimizer state
A100 PCIE 80GB$0.824/GPU-hour
The 28B is 55.3GB in weights alone, and a full 3B fine-tune penciled out as bf16 weights plus gradients plus fp32 master weights and two moments lands above 48GB — 80GB on one card avoids sharding entirely.

03 —

Getting PaliGemma 2 running from scratch

Four steps: pull the weights, run one inference, decode the boxes, serve it with vLLM.

01
Spin up the box, install deps, authenticate to the Hub
Pick an RTX 3090 24GB in the NexGPU console, boot the prebuilt PyTorch image, and get in over SSH or Jupyter. PaliGemma 2 landed in transformers 4.47, so do not run an old version. The repos are gated: accept the Gemma terms on the Hugging Face model page first, then log in with a token, or downloads will 403.
```
pip install -U 'transformers>=4.47' accelerate pillow && huggingface-cli login
```
02
Load in bf16 and run the first single-turn call
Use PaliGemmaForConditionalGeneration with PaliGemmaProcessor. The prompt must end with a newline — that character is part of the input format the model was trained on, and the processor adds it for you, which is exactly why you should go through the processor instead of hand-assembling input_ids. Tasks switch by prefix: cap {lang}, describe {lang}, ocr, answer {lang} {question}, detect {object}, segment {object}, with multiple objects separated by semicolons.
```
model = PaliGemmaForConditionalGeneration.from_pretrained('google/paligemma2-3b-mix-448', torch_dtype=torch.bfloat16, device_map='auto').eval()
```
03
Turn detect output back into pixel coordinates
The detect prefix does not return JSON. It returns a run of <loc0000>–<loc1023> tokens followed by a label. Four per box, in the order ymin, xmin, ymax, xmax, on a normalised 0–1023 grid: divide by 1024, then multiply y by image height and x by image width. Segment adds 16 <seg000>–<seg127> codewords that need the companion VQ-VAE decoder to become a mask — that decoder step is the one most tutorials skip and the one self-hosters get stuck on.
```
vals = [int(v) / 1024 for v in re.findall(r'<loc(\d{4})>', out)]  # ymin, xmin, ymax, xmax
```
04
Serve it over HTTP with vLLM
PaliGemmaForConditionalGeneration is explicitly on the vLLM supported-models list, covering both PaliGemma and PaliGemma 2, with LoRA and pipeline parallelism supported. Because this is a single-turn, single-image, short-output model, a max-model-len of 2048 is plenty — 1024 of it is image tokens at 448, and the rest covers the prefix and the answer. For the 10B tier, put it straight onto an RTX 5090 32GB.
```
vllm serve google/paligemma2-10b-mix-448 --dtype bfloat16 --max-model-len 2048
```

What a real labelling run actually costs

Say you need 100,000 receipt images turned into structured labels. Rent one RTX 3090 24GB ($0.193/GPU-hour) and run paligemma2-3b-mix-448: 10 hours of LoRA fine-tuning on your own 8,000 annotated samples, then 20 hours of batch inference producing detect + ocr labels. Compute = (10 + 20) × $0.193 = $5.79. Storage on a 30GB volume (6.1GB of weights plus dataset plus outputs) held for half a month = 30 × $0.414 × 0.5 = $6.21. Egressing 2GB of results = 2 × $0.0081 = $0.016. Total roughly $12.02 — and note that storage costs more than compute here, because NexGPU meters compute per second and stops billing the moment the instance stops, while storage keeps accruing until you destroy the volume. So destroy it when you are done. If accuracy pushes you to the 10B on an RTX 5090 32GB ($0.723/GPU-hour), the same 30 hours is 30 × $0.723 = $21.69. No minimum rental, no setup fee, no quota request.

04 —

FAQ

What is the minimum VRAM to run PaliGemma locally? Is 16GB enough?

On weights alone, PaliGemma 2 3B is 6.1GB in bf16, so a 16GB card holds it fine. The real trap is precision, not capacity: Google ships bf16 only, and neither the Tesla T4 (Turing) nor the Tesla V100 (Volta) has native bf16, while Gemma-family activations tend to blow up to NaN in fp16. So the minimum viable config is not '16GB', it is 'a card with native bf16'. The cheapest such card on NexGPU is the RTX 3090 24GB at $0.193/GPU-hour — less than the T4's $0.298 and with 8GB more memory.

I downloaded paligemma2-3b-pt-224 and it only outputs gibberish or repeats the prompt. Is the model broken?

It is not broken, you grabbed the wrong checkpoint. pt is a pretrained base that Google states is intended only for further fine-tuning; the ones you can prompt directly are mix (multi-task tuned) and ft (task-specific). Swap the ID to google/paligemma2-3b-mix-448 and it behaves. Also remember that the trailing newline in the prompt is part of the trained format — always go through PaliGemmaProcessor rather than assembling tokens yourself. Want to pull both and compare side by side? Half an hour on a NexGPU 3090 costs about ten cents.

Is there a GGUF for PaliGemma? Can I run it with llama.cpp or Ollama?

There is no official GGUF, and the Hub carries only a couple of community conversions of the original PaliGemma with three-digit download counts — effectively nobody runs that path. The quantisation routes that do work: bitsandbytes 4-bit (NF4), torchao int4_weight_only (the official transformers docs demo it on the 28B with group_size=128), and mlx-community 4-bit builds on Mac. The 3B at int4 is roughly 1.6GB, but the payoff is small when the bf16 original is only 6.1GB — rather than fight quantisation, just run native bf16 on NexGPU at $0.193/GPU-hour.

PaliGemma or Gemma 3 / Gemma 4 vision — which should I use? Is PaliGemma still worth it?

The split is clean. For conversation, multiple images, long context and general understanding, use Gemma 3 or Gemma 4 — they are natively multimodal, and PaliGemma is explicitly not a multi-turn chat model. But if your goal is one visual task done well and cheaply — ID field extraction, shelf product detection, medical image description — PaliGemma 2's transfer-first design, built-in loc/seg tokens and small 3B footprint remain the better starting point. The DOCCI fine-tune being the family's most-downloaded checkpoint is the evidence. Spin up two NexGPU boxes, try both, bill by the second, keep the winner.

What does <loc0512> mean in detect output, and how do <seg###> tokens become a mask?

<locXXXX> are 1024 position tokens reserved in the vocabulary; four of them make a box, in the order ymin, xmin, ymax, xmax, on a normalised grid — divide by 1024, then multiply y by image height and x by image width. Segment emits 16 further <seg000>–<seg127> codewords after the position tokens; those are VQ-VAE codebook indices and need the companion decoder to become a mask. That decoder step is where self-hosters most often stall, so get the decoder and your post-processing script working end to end in one sitting on a rented box. NexGPU gives you SSH, Jupyter, a web terminal and a REST API — debug however you like.

Can PaliGemma be used commercially? What is the licence?

The licence is the Gemma Terms of Use, not Apache or MIT. Commercial use is permitted, but the terms carry use restrictions and obligations that follow redistribution, and the Hub repos are gated — you must accept the terms and authenticate with a token, so a bare wget in CI gets a 403. Have counsel read the actual terms before production. As for where it runs: NexGPU spans 51 countries and regions, 1,175 verified rentable nodes and 2,498 GPUs, and your weights and data stay inside your own instance. We just supply the machines.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.