Skip to main content
NexGPU

Vision-Language Model · Image Understanding

Self-host BLIP: from the 247M captioning workhorse to billion-parameter BLIP-2 VQA

BLIP is the bedrock of open-source image labelling — Stable Diffusion dataset captioning, CLIP Interrogator prompt inversion, image-text retrieval reranking all run on it underneath. The weights fit on any consumer card. The real friction is picking the right generation and dodging a handful of well-known traps.

Sign up freeView pricing

BLIP (Bootstrapping Language-Image Pre-training) came out of Salesforce AI Research. It bootstraps clean web image-text pairs with a captioner-plus-filter loop, and one set of weights covers captioning, visual question answering and image-text retrieval. Its claim to fame is not a leaderboard spot but ubiquity as a pipeline part: `Salesforce/blip-image-captioning-base` pulls over 2.05M downloads a month, `blip-image-captioning-large` another 484K. pharmapsychotic's CLIP Interrogator chains BLIP with CLIP for prompt inversion and documents about 6.3GB of VRAM at default settings, dropping to roughly 2.7GB after `config.apply_low_vram_defaults()`.

One thing to get straight up front: the original `salesforce/BLIP` repository on GitHub was archived read-only on 3 March 2026, with the front page marked DEPRECATED and an explicit warning against using it for production or sensitive purposes — that code was last validated against PyTorch 1.10. This does not mean BLIP is dead. The weights are distributed normally on Hugging Face, and maintenance moved to LAVIS (BSD-3-Clause) and to the HF Transformers task classes `BlipForConditionalGeneration`, `BlipForQuestionAnswering` and `BlipForImageTextRetrieval`. Stop cloning the original repo from a 2022 README.

The family line has moved on several generations. BLIP-2 (MIT) bridges a frozen ViT and a frozen LLM with a 12-layer Q-Former, squeezing visual information through just 32 query tokens. InstructBLIP adds visual instruction tuning on top. xGen-MM — released publicly as BLIP-3, Apache 2.0 — swaps in a Phi-3 Mini backbone and handles interleaved multi-image input. The newest line, BLIP3o-NEXT (arXiv 2510.15857), fuses autoregression with diffusion for image generation on a Qwen3 backbone trained with GRPO. VRAM needs differ by an order of magnitude across those generations, so picking the wrong card is not a speed problem — it simply will not load.

01 —

BLIP family: variants and real VRAM footprints

From 247M to 12.2B parameters — nearly a 50x spread under one name

VersionParametersVRAMContextNotes
Salesforce/blip-image-captioning-base~247M (ViT-B/16)fp32 weights 0.99GB / fp16 ~0.5GB; 3–4GB is plenty for inference384×384 image + short textThe 2.05M-downloads-a-month batch captioning workhorse. Fast, near-zero VRAM, and the default behind kohya_ss and ComfyUI tagging nodes.
Salesforce/blip-image-captioning-large469,733,436 (ViT-L/16, 24-layer / 1024-dim vision tower)fp32 safetensors 1.88GB / fp16 ~0.94GB384×384 image, patch size 16Richer, more specific captions. Watch out: `max_length` in the config defaults to 20, so without an explicit `max_new_tokens` you get half a sentence.
blip-vqa-base / blip-itm-large-coco384,672,572 / ~446Mfp32 1.54GB / 1.78GB; halve for fp16384×384, size_divisor 32VQA handles single-image questions; ITM scores image-text matches for retrieval reranking. Normalisation uses mean [0.481, 0.458, 0.408] — do not substitute the ImageNet constants.
BLIP-2 opt-2.7b / flan-t5-xl3,744,761,856 / 3.94Bfp16 7.21GB, int8 3.61GB, int4 1.8GB (fp32 needs 14.43GB)Q-Former with 32 query tokens, 256-dim image-text fusionMIT licensed and commercially usable. Frozen ViT plus frozen LLM with only the Q-Former trained; beats Flamingo-80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters.
BLIP-2 flan-t5-xxl / InstructBLIP vicuna-13b12,229,596,672 / ~13Bfp16 ~24.5GB / ~26GB — needs a 32GB-plus cardInstruction-following long-form outputThe most articulate tier in the family. flan-t5-xxl is MIT, but vicuna-7b/13b carry an 'other' research-only licence via their LLaMA lineage; for commercial work use flan-t5 or InstructBLIP-flan-t5-xl (4,022,969,088 params, MIT).
xGen-MM (BLIP-3) / BLIP3o-NEXT-SFT-3B4,359,257,219 / 4,817,060,361bf16 weights 8.72GB / 9.67GBInterleaved multi-image input / image generationThe branch still under active maintenance, all Apache 2.0. xgen-mm-phi3-mini-interleave-r-v1.5 averages 65.1 on single-image benchmarks and 75.1 on QBench-2 multi-image (vs 52.4 non-interleaved); BLIP3o-NEXT uses a `blip3oQwenForCausalLM` architecture on a Qwen3 backbone, combining autoregression with diffusion.

02 —

Pick a card by workload

Do not waste a big card on tiny weights, and do not force 24.5GB onto a 24GB card — mismatched VRAM is a straight OOM

  • BLIP v1 batch captioning / VQA / retrieval (fp16, base or large)

    RTX 3090 24GB$0.193/GPU-hr

    Weights are under 1GB, so 24GB lets you push batch size high enough to amortise preprocessing — and this is the cheapest modern-architecture card per hour on our network.

  • BLIP-2 opt-2.7b / flan-t5-xl and xGen-MM 4B interactive inference

    RTX 4090 24GB$0.540/GPU-hr

    7.21GB fp16 or 8.72GB bf16 leaves most of the 24GB for KV cache and image tokens, with noticeably lower single-request latency than a 3090.

  • BLIP-2 flan-t5-xxl and InstructBLIP vicuna-13b at full precision

    RTX A6000 48GB$0.817/GPU-hr

    fp16 weights alone are 24.5GB; add roughly 500 placeholder tokens of activation per image and a 32GB card runs to the edge. 48GB is the tier where you stop retuning batch size.

  • Full fine-tuning of BLIP-2 (Adam fp16 optimiser state ~28.86GB) or BLIP3o-NEXT training

    A100 SXM4 80GB$1.088/GPU-hr

    Training needs roughly four times inference memory; 80GB plus NVLink fits opt-2.7b whole without gradient checkpointing, and a single node takes up to 14 GPUs for data-parallel scale-out.

03 —

BLIP running in ten minutes

Boot a prebuilt PyTorch image and get to your first caption without compiling anything

  1. 01

    Spin up an instance, install deps, sidestep the transformers v5 trap

    At console.nexgpu.net, launch an RTX 3090 24GB on the prebuilt PyTorch image and connect over SSH or Jupyter. Important: transformers v5 removed the `image-to-text` pipeline, so the many 2023-era BLIP tutorials that copy `pipeline("image-to-text", ...)` will error out — load the task class directly instead. `BlipModel` is also deprecated; use `BlipForConditionalGeneration`, `BlipForQuestionAnswering` or `BlipForImageTextRetrieval` depending on the job.

    pip install transformers accelerate pillow && export HF_HOME=/workspace/hf-cache
  2. 02

    Load the weights in fp16

    The large checkpoint is 1.88GB in fp32 and under 1GB once cast to fp16, so it loads in seconds. Point HF_HOME at a persistent volume and you skip the download next boot — BLIP weights are small enough that caching a copy costs essentially nothing in storage.

    model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large", dtype=torch.float16).to("cuda")
  3. 03

    Generate captions, and always set max_new_tokens explicitly

    This is the number one 'is the model broken?' moment with BLIP: config.json hardcodes `max_length` to 20, so with no override you get a truncated fragment. Pass `max_new_tokens` explicitly and pair it with beam search for stable quality. For batch runs, feed the whole image list to the processor at once so the GPU stays saturated.

    out = model.generate(**inputs, max_new_tokens=48, num_beams=3); print(processor.batch_decode(out, skip_special_tokens=True)[0])
  4. 04

    Step up to BLIP-2 for real VQA, quantise if VRAM is tight

    opt-2.7b is 3.74B parameters and 7.21GB in fp16. On a smaller card, int8 brings it to 3.61GB and int4 to 1.8GB with a single bitsandbytes config. Note that since transformers 4.46 the BLIP-2 processor asks you to set `processor.num_query_tokens` and expands the text with roughly 500 `<image>` placeholders per image — never let that text get truncated, or merging the embeddings will fail outright.

    model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", quantization_config=BitsAndBytesConfig(load_in_4bit=True), device_map="auto")

What a real captioning run costs

BLIP inverts the usual LLM cost structure: the weights are negligible and the dataset is the expensive part. The fp16 blip-image-captioning-large checkpoint is under 1GB, so storing it for a full month at $0.414/GB-month costs about $0.41. What you actually pay for is GPU time and the images themselves. Work it through: an RTX 3090 24GB is $0.193/GPU-hr, so six hours of batch captioning is 0.193 × 6 = $1.158. Want a second, finer pass over the same set with BLIP-2 flan-t5-xxl? Its fp16 weights are around 24.5GB, so move to an RTX A6000 48GB: $0.817/GPU-hr × 2 hours = $1.634. Both passes together come to $2.79. The line to actually watch is storage: 200GB of source images sits at $0.414/GB-month × 200 = $82.80/month, thirty times the compute bill. Compute billing stops the second the instance stops; storage keeps accruing until the volume is destroyed — so the right pattern for BLIP work is to export results immediately (egress $0.0081/GB × 200 = $1.62) and then destroy the volume. Everything is metered per second, with no minimum, no setup fee, and no quota request.

04 —

FAQ

How much VRAM does BLIP actually need? Is an 8GB card enough?

For BLIP v1, more than enough — with room to spare. The base model is roughly 247M parameters at 990MB in fp32, half that in fp16, and runs comfortably in 3–4GB. The large model is 469.7M parameters and under 1GB in fp16. For a real-world reference point, CLIP Interrogator (BLIP chained with CLIP) documents about 6.3GB at default settings and about 2.7GB in low-VRAM mode. BLIP-2 is a different story: opt-2.7b needs 7.21GB in fp16, which is painfully tight on an 8GB card, so you would lean on int8 (3.61GB) or int4 (1.8GB). If you would rather not fight OOM, a NexGPU RTX 3090 24GB is $0.193/GPU-hr.

The salesforce/BLIP GitHub repo is archived. Can I still use BLIP, and what should I migrate to?

The original repo went read-only on 3 March 2026, marked DEPRECATED with a warning against production or sensitive use, and its code was last validated on PyTorch 1.10. The weights themselves are perfectly fine — the Salesforce/blip-* checkpoints are still distributed on Hugging Face under BSD-3-Clause. Two migration paths: LAVIS, Salesforce's multimodal library (also BSD-3-Clause, covering BLIP, BLIP-2, InstructBLIP and X-InstructBLIP), or the three HF Transformers task classes directly. On a NexGPU prebuilt PyTorch image, the latter is one `pip install transformers` away.

Why do BLIP captions always cut off mid-sentence?

The model is fine — config.json hardcodes `max_length` to 20. Pass `max_new_tokens=48` (or whatever length you need) to generate, and add `num_beams=3` for more stable output. Nearly everyone hits this once on their first run. While you are at it: transformers v5 removed the `image-to-text` pipeline entirely, so `pipeline("image-to-text", ...)` from older tutorials will error — load `BlipForConditionalGeneration` directly. Rent a NexGPU instance, get the script right once, and reuse it forever.

How do BLIP, BLIP-2, InstructBLIP and BLIP-3 relate, and which should I pick?

BLIP v1 (2022) is a small end-to-end model, 247M–470M parameters, focused on captioning, VQA and retrieval — fast and cheap. BLIP-2 (2023, MIT) introduces a 12-layer Q-Former that pushes frozen-ViT features through 32 query tokens into a frozen LLM; opt-2.7b totals 3.74B parameters and gives you genuine open-ended question answering. InstructBLIP adds visual instruction tuning on top. xGen-MM (BLIP-3, Apache 2.0) moves to a Phi-3 Mini backbone with interleaved multi-image support at 4.36B parameters and 8.72GB of bf16 weights. BLIP3o-NEXT pivots to image generation. The rule is simple: captions only, use v1; question answering, use BLIP-2; multi-image reasoning, use xGen-MM. On NexGPU those map to RTX 3090 24GB at $0.193, RTX 4090 24GB at $0.540, and RTX 4090 24GB at $0.540.

BLIP-2 warns about num_query_tokens and then fails merging embeddings. How do I fix it?

This is a behaviour change introduced in transformers v4.46. BLIP-2 and InstructBLIP now expect a `num_query_tokens` attribute on the processor (read it from `model.config.num_query_tokens`, default 32) and expand the input text with that many `<image>` placeholders — in practice around 500 tokens per image. If your tokenizer has a shorter truncation length, those placeholders get cut and merging the vision and text embeddings crashes. Set the attribute and make sure the text is not truncated. NexGPU's web terminal and Jupyter both give you the full traceback, and support is bilingual over Telegram with no ticket queue.

Can I use BLIP models commercially? What are the licences?

Three cases. All of BLIP v1 (captioning-base/large, vqa-base, itm-*) is BSD-3-Clause and fine commercially. The BLIP-2 opt and flan-t5 series plus InstructBLIP-flan-t5-xl (4.02B parameters) are MIT, also fine. Be careful with InstructBLIP-vicuna-7b and -13b: they carry an 'other' research-only licence through their Vicuna/LLaMA lineage, so use the flan-t5 branch for commercial work. xGen-MM (BLIP-3) and the whole BLIP3o line are Apache 2.0, the most permissive of the lot. Once you have picked a variant, validating it on NexGPU costs a few cents — per-second billing, and compute charges stop the moment you stop the instance.

More in Multimodal vision models

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.