Multimodal vision-language model

CogVLM self-hosting: from a 16GB single-card Int4 build to the 42GB BF16 reality

The visual-expert architecture makes CogVLM cost noticeably more VRAM than its parameter count suggests. The project published real numbers — here they are, mapped to cards you can actually rent.

CogVLM · self-hosted

CogVLM comes out of Tsinghua KEG and Zhipu. Both the code and the weights now live under the zai-org namespace; the old THUDM/* paths on Hugging Face redirect to zai-org/*, so either spelling still pulls the weights. Architecturally it is not the usual "project image features into the text sequence" VLM: it bolts a separate set of visual-expert weights onto every transformer layer. The original CogVLM-17B is 10B vision parameters plus 7B language parameters on a Vicuna-7B-v1.5 base; CogVLM2 swaps in Meta-Llama-3-8B-Instruct and lands at 19B.

That design shows up directly on the VRAM bill. The official basic_demo table is unambiguous: for the 19B series, BF16 inference takes 42GB and Int4 inference takes 16GB, both measured at 2K dialogue length. cli_demo.py even hard-codes a check — if it sees less than 48GB of VRAM and you have not enabled quantization, it tells you to switch to the multi-GPU script or add a quant flag. Layer on the 1344×1344 input resolution and the pile of visual tokens it produces, and the prefill peak gets worse still. "How much VRAM does CogVLM need" has no single answer, only a table.

One more thing worth saying plainly: CogVLM2 stopped in May 2024, and the successor line lives in the zai-org/GLM-V repo — GLM-4.1V-9B-Thinking (2025-07-01, 64K context), GLM-4.5V (2025-08-11), then GLM-4.6V and GLM-4.6V-Flash (2026-04-02, 128K context, Flash under MIT). A greenfield project should start at GLM-4.6V-Flash, not at CogVLM2. But cogvlm2-llama3-chat-19B still pulls over seven thousand downloads a month, the grounding-generalist checkpoint is still kept around for box-drawing work, and nobody re-runs a whole eval suite just because a newer generation shipped. This page is for the people who still have to get it running.

01 —

Every CogVLM variant, with the VRAM numbers that were actually published

Figures come from the official basic_demo table and the model cards. Where the project published nothing, we leave the cell blank rather than guess.

Version	Parameters	VRAM	Context	Notes
cogvlm2-llama3-chinese-chat-19B	19B (Llama-3-8B-Instruct base + visual expert)	BF16 ~42GB / Int4 ~16GB	8K text	Bilingual Chinese-English image understanding and multi-turn dialogue, up to 1344×1344 input. The default pick for Chinese-language self-hosting.
cogvlm2-llama3-chat-19B (plus the official -int4 repo)	19B	BF16 ~42GB / Int4 ~16GB	8K text	English-only. The separate -int4 repo uses bitsandbytes 4-bit and its card states it must run on Linux with an NVIDIA GPU.
cogvlm2-video-llama3-chat / -base	19B	Same class as the 19B line, ~42GB BF16	2K text	Video variant: a fixed 24 frames at 224×224 each, single-turn only. Do not substitute it for the image model — the resolution is an order of magnitude lower.
cogagent-9b-20241220	9B (GLM-4V-9B base)	BF16 ~29GB / INT8 ~15GB / INT4 ~8GB	demo default max_length 4096	An agent execution model, not a chat model: it reads screenshots and emits click / type / scroll actions across Windows, macOS and Android at 1120×1120. The project explicitly discourages INT4 because quality drops noticeably.
cogvlm-chat-hf (original CogVLM-17B)	17B (10B vision + 7B language, Vicuna-7B-v1.5)	FP16 ~40GB / INT4 ~11GB	~4K, inherited from Vicuna-7B-v1.5	490×490 resolution, with base-224, base-490 and grounding-generalist alongside it. The contemporaneous CogAgent-18B is 11B vision + 7B language at 1120×1120 and ~12.6GB in INT4.
GLM-4.6V / GLM-4.6V-Flash (successor line, zai-org/GLM-V)	106B / 9B	No official VRAM table published; FP8 weights exist for the 106B	128K	Released 2026-04-02 and served natively by both vLLM and SGLang. Flash is MIT-licensed. Start new work here, not on CogVLM2.

02 —

Picking a card on NexGPU, per variant

The VRAM has to actually fit. A 42GB BF16 model does not shrink because you would prefer a cheaper card.

Int4 CogVLM2-19B (16GB), or original CogVLM-17B INT4 (11GB) / CogAgent-18B INT4 (12.6GB)
RTX 3090 24GB$0.193/GPU-hr
The project's own INT4 numbers were measured on exactly one RTX 3090 (24G), and 24GB leaves prefill headroom for the visual tokens a 1344×1344 image produces.
Full-precision BF16 cogvlm2-llama3-chinese-chat-19B on one card (42GB)
RTX A6000 48GB$0.817/GPU-hr
48GB is the threshold hard-coded into cli_demo.py: below it, unquantized, the script pushes you toward multi-GPU. At 48GB the script stays quiet and you never touch a device_map.
cogagent-9b-20241220 as a GUI agent (BF16, 29GB)
RTX 5090 32GB$0.723/GPU-hr
29GB into 32GB fits without dropping to the INT4 mode the project warns against — and this is the one member of the family vLLM serves natively, under GLM4VForCausalLM, so throughput is no longer bottlenecked on HF generate.
BF16 LoRA with the visual expert unfrozen (73GB per GPU, 8-way DeepSpeed ZeRO-2 officially)
A100 SXM4 80GB × 8$1.088/GPU-hr ($8.704/hr for eight)
The published run used 8×A100 80G with ZeRO-2, so copying the config is the cheapest path. NexGPU nodes go up to 14 GPUs, so all eight sit in one box and you skip cross-node NCCL tuning.

03 —

Getting CogVLM2 running on NexGPU

Four steps from a bare instance to an OpenAI-compatible endpoint. Step two is where most people lose an afternoon.

01
Start an instance whose VRAM matches the plan
In console.nexgpu.net pick the card: RTX 3090 24GB for Int4, RTX A6000 48GB for full BF16, RTX 5090 32GB for the GUI agent. Choose a prebuilt PyTorch image — there are 2,000+ of them, including vLLM, ComfyUI and Jupyter — and you get SSH, Jupyter, a web terminal, a REST API and a CLI out of the box. Billing is metered per second and priced per hour, with no minimum, no setup fee and no quota request.
02
Pin transformers first — this is the big one
CogVLM2 loads its own modeling file through trust_remote_code=True, and basic_demo/requirements.txt only says transformers>=4.40 with no upper bound. Install something recent and you get AttributeError: 'CogVLMForCausalLM' object has no attribute '_extract_past_from_model_output' — reproduced in the repo's issues on transformers 4.52.4. Newer GenerationMixin dropped that internal method while the remote modeling code still calls it. Pin below 4.45 before you debug anything else.
```
pip install "transformers<4.45" "bitsandbytes>=0.43.1" accelerate xformers torchvision timm einops sse-starlette
```
03
Pull the weights and choose a quantization tier
cli_demo.py's --quant accepts only 4 or 8, with 0 (no quantization) as the default; on a 24GB card use --quant 4. For multiple GPUs use cli_demo_multi_gpus.py, which calls infer_auto_device_map with no_split_module_classes=["CogVLMDecoderLayer"] and was tested on three 16GB cards at roughly 13–14.5GB each. Watch out: for two GPUs or fewer the script hard-codes a 16GiB per-device cap, so a BF16 run needs you to raise max_memory yourself or it will be sharded into a config that cannot hold the model. Separately, the original CogVLM ships no tokenizer of its own — load LlamaTokenizer from lmsys/vicuna-7b-v1.5.
```
huggingface-cli download zai-org/cogvlm2-llama3-chinese-chat-19B --local-dir ./cogvlm2-zh && python basic_demo/cli_demo.py --quant 4
```
04
Expose an endpoint, and decide about vLLM up front
basic_demo/openai_api_demo.py brings up an OpenAI-format service that existing frontends and gateways can hit directly; web_demo.py serves a chainlit UI. Do not go looking for CogVLM in vLLM, though — neither CogVLM nor CogVLM2 has ever appeared in its supported-architectures list. The only relatives vLLM serves are cogagent-9b-20241220 (GLM4VForCausalLM) and the GLM-4.1V / 4.5V / 4.6V line. Continuous batching and PagedAttention here mean changing models, not changing flags.
```
python basic_demo/openai_api_demo.py   # for throughput instead: vllm serve zai-org/cogagent-9b-20241220 --trust-remote-code
```

A real invoice: captioning twenty thousand images

Run cogvlm2-llama3-chinese-chat-19B in BF16 for bulk captioning on an RTX A6000 48GB at $0.817/GPU-hr. Boot, download the 19B BF16 weights (~38GB of safetensors) and install dependencies — call it 25 minutes: 0.42 × $0.817 ≈ $0.34. Six hours of batch inference: 6 × $0.817 = $4.90. Export 2GB of results at the $0.0081/GB median egress rate ≈ $0.02. Total ≈ $5.26. Prefer the Int4 tier? An RTX 3090 24GB is $0.193/GPU-hr, so the same six hours is 6 × $0.193 = $1.16 — under a quarter of the cost, at the price of quantization loss that is fine for triage and wrong for final output. And here is the comparison people get backwards: the official multi-GPU script was measured at only 13–14.5GB per card across three 16GB cards, which looks frugal, but three Tesla T4 16GB come to 3 × $0.298 = $0.894/hr — more than the single A6000 at $0.817/hr, plus a device_map to babysit. On this model, multi-GPU is not a savings strategy; it is what you do when nothing else fits. Finally, billing is metered per second and priced per hour, so compute charges stop the moment the instance stops — only storage keeps accruing until the volume is destroyed (median $0.414/GB-month, so ~60GB of weights and cache is $24.84/month). Delete the volumes you are done with.

04 —

FAQ

How much VRAM does CogVLM2-19B really need to run locally?

The official basic_demo table says 42GB for BF16 inference and 16GB for Int4, both measured at 2K dialogue length. cli_demo.py also hard-codes a 48GB warning line: below it, unquantized, it pushes you to multi-GPU. So a 24GB card is an Int4-only card, and full precision wants 48GB. On NexGPU those two tiers are RTX 3090 24GB at $0.193/GPU-hr and RTX A6000 48GB at $0.817/GPU-hr, billed per second — stop the instance the moment you are done.

Can I serve CogVLM with vLLM? Why does it fail to load?

Because it is not supported. vLLM's multimodal architecture list contains GLM4VForCausalLM (covering glm-4v-9b and cogagent-9b-20241220), Glm4vForConditionalGeneration (GLM-4.1V-Thinking) and Glm4vMoeForConditionalGeneration (GLM-4.5V) — but no CogVLM or CogVLM2 entry at all. CogVLM2 only runs through HF transformers with trust_remote_code, which means giving up continuous batching. If you need throughput, move to cogagent-9b-20241220 or GLM-4.6V-Flash. NexGPU's 2,000+ prebuilt images ship both vLLM and PyTorch, so spin up two instances side by side and measure before you commit.

Why do I get an error about _extract_past_from_model_output right after installing?

Your transformers is too new. The repo's requirements.txt says transformers>=4.40 with no ceiling; newer GenerationMixin removed that internal method while the remote modeling code still calls it, and the issue tracker has it reproduced on 4.52.4. Pin below 4.45. Related traps: 'Params4bit' object has no attribute '_parameters' when combining 4-bit with multi-GPU, and the default script only using a single card. Version archaeology is cheap when billing is per second — half an hour on a NexGPU RTX 3090 costs about ten cents.

Has CogVLM been superseded by GLM-4.6V? Is it still worth deploying?

The line did move on: the successor repo is zai-org/GLM-V, with GLM-4.1V-9B-Thinking (2025-07-01, 64K), GLM-4.5V (2025-08-11), and GLM-4.6V plus GLM-4.6V-Flash (2026-04-02, 128K, Flash under MIT). New projects should start there. But cogvlm2-llama3-chat-19B still sees over seven thousand downloads a month, purpose-built checkpoints like grounding-generalist have no drop-in replacement, and a tuned production pipeline does not re-qualify itself for free. If you want a real A/B, one NexGPU account can run CogVLM2 on an RTX A6000 and GLM-4.6V-Flash on an RTX 5090 at the same time and compare the outputs directly.

Are the CogVLM weights usable commercially?

The code is Apache-2.0; the weights are not. CogVLM weights carry a separate Model License — open for academic use, registration required for commercial use — and CogVLM2 weights are under the CogVLM2 LICENSE, with Meta's LLAMA3_LICENSE stacked on top for the Llama-3-based variants. The successor GLM-4.6V-Flash, by contrast, is MIT. Licensing is your legal team's call; on the compute side we add no gates: NexGPU is metered per second, no minimum, no setup fee, no quota request, with bilingual support over Telegram and no ticket queue.

How many GPUs do I need to fine-tune CogVLM2?

The published figure is 73GB per GPU for BF16 LoRA with the visual expert included, on 8×A100 80G with DeepSpeed ZeRO-2 at 2K training text length. Freezing the visual expert saves a lot, but at 19B this is not a consumer-single-card job. NexGPU's A100 SXM4 80GB is $1.088/GPU-hr, so eight of them is $8.704/hr — and with up to 14 GPUs per node and a maximum node VRAM of 2,152GB, all eight sit in one machine and you skip cross-node tuning entirely. The network spans 1,175 verified rentable nodes, 2,498 GPUs and 75 GPU models across 51 countries and regions.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.