Skip to main content
NexGPU

Vision-Language Model (VLM)

Running LLaVA locally: from a 4.37GB Q4 build to a 17.1GB OneVision-2, on one card

From llava-1.5-7b-hf — still pulling over 2.5 million downloads a month — to LLaVA-OneVision-2-8B released in April 2026, this lineage has changed repositories three times and licences twice. Here are the real VRAM numbers, the real commands, and the exact card to rent.

Sign up freeView pricing

LLaVA (Large Language and Vision Assistant) came out of work by Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee at UW–Madison and Microsoft Research. It was the first project to show that the minimalist recipe — vision encoder, a projection layer, a language model — was genuinely competitive. Its standing is still visible in the download counts: llava-hf/llava-1.5-7b-hf clears 2,521,243 downloads in a month, making it one of the most cited, most fine-tuned, most pipeline-embedded open VLMs in existence.

The catch is that almost every tutorial you will find stops at LLaVA-1.5 (2023) or LLaVA-1.6 (early 2024). The line has actually moved house three times. haotian-liu/LLaVA was home for the 1.5/1.6 era. Development then shifted to LLaVA-VL/LLaVA-NeXT, whose README now labels its own training pipeline legacy and points readers at lmms-engine. Active work happens in EvolvingLMMs-Lab/LLaVA-OneVision-1.5: the LLaVA-OneVision-1.5 technical report landed 2025-09-30 (arXiv 2509.23661), an RL recipe followed on 2025-12-11, the codec-aligned OneVision-Encoder on 2026-02-10, and LLaVA-OneVision-2 shipped from that same repo on 2026-04-30. So no, LLaVA is not dead — you just have to pull from the right repo.

One trap is specific to this family: what eats your VRAM is often not the weights, it is the visual tokens. LLaVA-1.5 uses CLIP ViT-L/14-336 and emits a fixed 576 visual tokens per image (336÷14=24, 24²=576). LLaVA-1.6 added AnyRes, supporting 672×672, 336×1344 and 1344×336; a 672×672 image is split into a 2×2 grid plus a global thumbnail, so 5×576 ≈ 2,880 tokens go straight into context. OneVision-2 then applies HEVC-style codec-aligned patch selection on video — keep I-frames dense, keep only motion- and residual-rich patches from P-frames — fitting 18 frames instead of 6 into the same token budget, roughly 3× the temporal coverage. Which means the same 8B model needs a completely different card depending on whether you feed it one photo or sixteen frames. Below, tier by tier, with a card you can boot on NexGPU right now.

01 —

Which LLaVA versions actually exist today, and what each one costs in VRAM

Parameter counts read from Hugging Face safetensors metadata; VRAM is weights only (bf16/fp16 at 2 bytes per parameter), excluding KV cache and visual-token activations.

VersionParametersVRAMContextNotes
LLaVA-OneVision-2-8B-Instruct (lmms-lab-encoder)8.53B (8,527,213,568)bf16 weights ≈ 17.1GBContext ceiling not published on the card; video side is set by num_frames × max_pixelsThe current flagship, released 2026-04-30. Qwen3-8B backbone plus the codec-aligned OneVision-Encoder, natively bilingual EN/ZH, architecture LlavaOnevision2ForConditionalGeneration. Needs transformers>=5.7.0, torch>=2.4 and trust_remote_code=True; the codec video backend additionally wants ffmpeg 4.4.x–7.x. Apache-2.0.
LLaVA-OneVision-2-4B-Instruct4B (Qwen3-4B-Instruct-2507 backbone)Weights not released — no estimate givenSame OneVision-2 familyThe official README marks this coming soon and the weights are not downloadable from Hugging Face yet. If a blog post claims the 4B is out, skip it rather than burn GPU-hours on a 404. If you need that tier today, use LLaVA-OneVision-1.5-4B-Instruct.
LLaVA-OneVision-1.5-8B-Instruct (plus 8B-RL)8.53B (8,527,214,624)bf16 weights ≈ 17.1GBNot published on the cardShipped September 2025 with a DeepGlint-AI/rice-vit-large-patch14-560 vision tower and a Qwen language side. The selling point is native-resolution processing plus a genuinely open training pipeline — the 85M-sample mid-training corpus was released alongside the weights. An RL variant, mvp-lab/LLaVA-OneVision-1.5-8B-RL, followed in December 2025 at an identical parameter count. Apache-2.0.
LLaVA-OneVision-1.5-4B-Instruct4.74B (4,741,610,528)bf16 weights ≈ 9.5GBNot published on the cardThe most comfortable tier for a single 24GB card: 9.5GB of weights leaves a dozen-plus GB entirely for KV cache and multi-image tokens. For bulk captioning, product-image tagging, or invoice and ID field extraction, this beats forcing the 8B onto the same card. Apache-2.0.
llava-onevision-qwen2-7b-ov-hf / -72b-ov-hf (2024 OneVision)8.03B (8,030,807,584) / 73.18B (73,181,570,592)fp16 ≈ 16.1GB / ≈ 146GBRequires transformers >= 4.45The tier with native transformers support and no trust_remote_code needed — architecture LlavaOnevisionForConditionalGeneration, recognised directly by vLLM. Note that the 72B's fp16 weights come to about 146GB, more than a single H200's 141GB, so one card physically cannot hold it: tensor-parallel or drop precision. Apache-2.0.
LLaVA-1.6 / LLaVA-NeXT and LLaVA-1.5 (previous generation)7B / 13B / 34B1.5-7B fp16 ≈ 14.1GB, 1.5-13B fp16 ≈ 26.7GB; 1.6-7B GGUF: Q4_K_M 4.37GB, Q5_K_M 5.13GB, Q8_0 7.7GBllava:7b is 32K; llava:13b and llava:34b are 4K (per Ollama)The generation with the deepest tooling support — Ollama's llava:7b (4.7GB), llava:13b (8.0GB) and llava:34b (20GB) are all 1.6. Licensing is the trap here: LLaVA-1.5 weights carry the Llama 2 Community License, while llava-v1.6-mistral-7b-hf (built on Mistral-7B-Instruct-v0.2) is the Apache-2.0 one. Check which you actually pulled before shipping commercially.

02 —

Pick by workload — do not rent an H100 to caption one photo

NexGPU spans 51 countries and regions, 1,175 verified rentable nodes, 2,498 GPUs across 75 models, up to 14 GPUs per node and 2,152GB max node VRAM. Rates below are list price per GPU-hour.

  • Get LLaVA-1.6-7B running on Q4 first — quality comparison and prompt iteration

    RTX 3090 24GB$0.193/GPU-hr

    Q4_K_M weights are 4.37GB plus the mmproj-model-f16 vision projector; what remains of 24GB is plenty to run AnyRes at full resolution. It is also the cheapest 24GB card on the network — a third less than the 16GB Tesla T4 at $0.298.

  • LLaVA-OneVision-1.5-4B in bf16 as a resident bulk captioning / tagging API

    RTX 4090 24GB$0.540/GPU-hr

    9.5GB of weights leaves a dozen-plus GB for vLLM's KV cache and multi-image tokens, and per-request latency is clearly better than a 3090. This is the sweet spot for something you leave running and serve traffic from.

  • LLaVA-OneVision-2-8B bf16 online serving, or 16-frame video QA

    RTX 5090 32GB$0.723/GPU-hr

    After 17.1GB of weights you still have roughly 14GB for visual tokens and concurrency. In multi-frame work it is never the weights that blow up first, it is the activations — a 24GB card OOMs the moment you raise num_frames, and 32GB is the safe line.

  • OneVision-72B fp16 full-precision evaluation, or full-parameter / large-scale LoRA on 8B

    A100 SXM4 80GB (two-way tensor parallel)$1.088/GPU-hr

    73.18B in fp16 is about 146GB — more than a single H200's 141GB, so one card is physically out. Two 80GB cards is the floor. With up to 14 GPUs per node on NexGPU, scaling up to full-parameter fine-tuning later does not mean rebuilding your environment elsewhere.

03 —

Four steps to a running LLaVA

Built around the current LLaVA-OneVision-2-8B-Instruct, with a GGUF route at the end for smaller cards.

  1. 01

    Spin up an instance on a vLLM prebuilt image

    At console.nexgpu.net, pick a card by VRAM (RTX 5090 32GB or better for the 8B in bf16) and choose a PyTorch or vLLM image from the 2,000+ prebuilt options so you skip CUDA and driver setup entirely. Get in over SSH, Jupyter or the web terminal. OneVision-2 is version-sensitive, so pin the dependencies first — anything below transformers 5.7.0 will fail outright when the custom modelling code loads.

    pip install -U vllm "transformers>=5.7.0" "torch>=2.4" pillow requests decord
  2. 02

    Pull the weights and serve an OpenAI-compatible endpoint

    One vLLM command downloads the model and stands up the HTTP server. OneVision-2 ships custom modelling code, so trust_remote_code is not optional. If you are instead running the 2024-era llava-onevision-qwen2-7b-ov-hf, transformers supports it natively and you can drop that flag. Expect a few minutes of downloading 17GB before first token — that time is billed per second, and compute billing stops the moment you stop the instance.

    vllm serve lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct --trust-remote-code --dtype bfloat16 --port 8000
  3. 03

    Send your first image over the OpenAI protocol

    Once it is up it is plain /v1/chat/completions, mixing image_url and text entries in the content array. The most common failure here is setting max_model_len too low: a single 672×672 image under AnyRes can consume nearly 2,900 visual tokens, and with a system prompt on top you hit the ceiling fast — after which you get the error about image features not matching the number of image tokens.

    curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct","messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"https://example.com/a.jpg"}},{"type":"text","text":"How many people are in this image, and what is each doing?"}]}]}'
  4. 04

    Low-VRAM route: GGUF plus llama.cpp

    On 16GB or less, quantise. cjpais/llava-1.6-mistral-7b-gguf carries the full ladder from Q3_K_XS at 2.99GB to Q8_0 at 7.7GB. Note that the vision half lives in a separate mmproj-model-f16.gguf and must be loaded with --mmproj; downloading only the language weights will not work. Also be aware that llama.cpp folded the old llava-cli into the unified libmtmd — the entry point today is llama-mtmd-cli, LLaVA is classified there as a legacy architecture, and the mtmd sub-project states it is under very heavy development with breaking changes expected, so pin your version before production. For a one-liner, ollama run llava:7b (4.7GB) also works.

    llama-mtmd-cli -m llava-v1.6-mistral-7b.Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image ./a.jpg -p "Describe this image"

A real LoRA fine-tune, costed out

Say you want LLaVA-OneVision-1.5-8B-Instruct adapted to your own support-ticket screenshots. Take an A100 PCIE 80GB at $0.824/GPU-hr and run single-card LoRA for 12 hours: 12 × $0.824 = $9.89. Data plus weights occupy 80GB of storage (17.1GB of weights, dataset, checkpoints); at the $0.414/GB-month median that is 80 × $0.414 × 3 ÷ 30 = $3.31 for three days. Pulling the LoRA adapter back down is 0.5GB × $0.0081/GB = $0.004. Total: roughly $13.2. For perspective, the LLaVA-OneVision-1.5 team disclosed that training the 8B cost them around $16,000 (reckoned at roughly $0.60 per A100 GPU-hour) — you do not have to repeat that; standing on their Apache-2.0 weights and fine-tuning costs two figures. Cheaper still: just validating quality with Q4-quantised LLaVA-1.6-7B on an RTX 3090 24GB for two hours is 2 × $0.193 = $0.386, less than a coffee. At the top end: OneVision-72B in fp16 needs two A100 SXM4 80GB cards, so a six-hour evaluation run is 2 × $1.088 × 6 = $13.06. Everything is metered per second and priced per hour — no minimum, no setup fee, no quota request. Compute billing stops when the instance stops; storage runs until you destroy it.

04 —

FAQ

Is LLaVA obsolete? Is it still worth self-hosting in 2026?

The old versions are obsolete; the project is not. LLaVA-1.5 (2023) and LLaVA-1.6/NeXT (2024) really are previous-generation — the LLaVA-VL/LLaVA-NeXT repo labels its own training pipeline legacy. But the mainline kept moving: LLaVA-OneVision-1.5 on 2025-09-30, an RL recipe on 2025-12-11, the codec-aligned OneVision-Encoder on 2026-02-10, and LLaVA-OneVision-2 on 2026-04-30, all out of the EvolvingLMMs-Lab repo. Its pitch also shifted from 'highest benchmark score' to 'genuinely open end to end' — weights, the 85M mid-training corpus, instruction data and training code, all Apache-2.0. If you want a VLM you can audit, reproduce and re-architect, this line is still the least painful starting point, and validating it on a NexGPU card takes minutes.

How much VRAM does LLaVA actually need? Give me the numbers.

Weights only: LLaVA-1.6-7B Q4_K_M is 4.37GB and Q8_0 is 7.7GB; llava-1.5-7b-hf in fp16 is about 14.1GB and the 13B about 26.7GB; LLaVA-OneVision-1.5-4B in bf16 is about 9.5GB; both OneVision-1.5-8B and OneVision-2-8B are 8.53B parameters, roughly 17.1GB in bf16; OneVision-72B in fp16 is about 146GB. Then add headroom for how you actually use it: 30% for single-image inference, at least 100% for multi-frame video or high-resolution AnyRes. Mapped to rentals: Q4 experimentation on an RTX 3090 24GB ($0.193/GPU-hr), a resident 4B on an RTX 4090 24GB ($0.540), 8B online serving on an RTX 5090 32GB ($0.723), and 72B on two A100 SXM4 80GB ($1.088). All billed per second, so getting it wrong costs cents.

Why do I get 'image features and image tokens do not match' as soon as I run it?

This is the classic LLaVA-family error, and it almost always traces to processor configuration or context truncation. Since 4.46, transformers explicitly warns you to set patch_size, num_additional_image_tokens and vision_feature_select_strategy on the processor — only with those can it work out how many <image> placeholders to expand per image (LLaVA-1.5 is a fixed 576 per image; AnyRes multiplies that several times over). If the text gets truncated, the placeholder count no longer matches the visual features and merging the embeddings throws. While you are there, remember the other one: set processor.tokenizer.padding_side = "left" for batched generation or your outputs quietly degrade. Rebuilding environments locally to chase this is slow; on NexGPU you swap prebuilt images and relaunch in under a minute.

Can I use LLaVA commercially? How does the licensing work?

There are two buckets, and you must check which repo you pulled from. The LLaVA-1.5 weights (llava-hf/llava-1.5-7b-hf, 13b-hf) carry the Llama 2 Community License with its extra conditions. Meanwhile llava-v1.6-mistral-7b-hf (built on Mistral-7B-Instruct-v0.2), the llava-onevision-qwen2 series, the whole LLaVA-OneVision-1.5 family including 8B-RL, and LLaVA-OneVision-2-8B-Instruct are all Apache-2.0 — clean for commercial use. The original haotian-liu/LLaVA code is Apache-2.0, but its weights remain bound by base-model and training-data terms. To sidestep the question entirely, start from OneVision-1.5 or OneVision-2 — and on NexGPU you can pull both onto one instance and compare them side by side.

How do I deploy LLaVA-OneVision-2 for video, and how many GPUs does it take?

Video is exactly where OneVision-2 improved: the codec-aligned encoder selects patches the way HEVC does — dense patches from I-frames, only motion- and residual-rich regions from P-frames — covering 18 frames instead of 6 within the same token budget, about 3× the temporal reach. Deployment needs transformers>=5.7.0 and decord, and the codec backend additionally wants codec-video-prep, opencv-python and ffmpeg 4.4.x–7.x. The official starting configuration is num_frames=16 with max_pixels=200704. On memory, 17.1GB of weights is the floor and 16 frames of visual tokens on top make a 24GB card very tight — start at an RTX 5090 32GB ($0.723/GPU-hr). For longer clips or real concurrency go straight to an RTX A6000 48GB ($0.817) or A100 PCIE 80GB ($0.824); all three boot on demand at NexGPU.

LLaVA versus Qwen-VL or InternVL — which should I pick?

If you only want the highest-scoring VLM that works out of the box, there are easier choices. LLaVA's advantage is elsewhere: the simplest architecture in the field (vision tower, projector, LLM — you can read it in one sitting and modify it), the most transparent training pipeline (OneVision-1.5 open-sourced the 85M mid-training corpus and the instruction data alongside the weights), and the deepest downstream ecosystem (llava-hf/llava-1.5-7b-hf alone sees 2.5 million downloads a month, and it is the baseline in half the papers you will read). For research, custom architectures, or regulated models where you must account for your data provenance, this line is the first pick. All of which needs real hardware to settle — NexGPU has 75 GPU models and 2,498 cards across 51 countries and regions, per-second billing with no minimum, and bilingual Telegram support with no ticket queue. Benchmarking all three on the same card costs about as much as lunch.

More in Multimodal vision models

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.