Multimodal document vision-language model

Kosmos-2.5 self-hosted: 1.3B params that read a full page into coordinate-tagged Markdown

From Microsoft's unilm, MIT-licensed, under 3GB in bf16 — one RTX 3090 turns scans, receipts and paper pages into structured text, from $0.193/GPU-hour.

Kosmos · self-hosted

Kosmos is a multimodal line inside Microsoft's unilm project, and its four generations do genuinely different jobs. Kosmos-1 shipped a paper and no weights. Kosmos-2 wrote grounding into the language model, emitting Markdown-style links like [text span](boxes) (microsoft/kosmos-2-patch14-224, 1.6B). Kosmos-2.5 pivoted to the part people actually pay for — machine reading of text-intensive images (microsoft/kosmos-2.5, 1.3B). Kosmos-2.5-chat adds document QA on top. All of it is MIT, all of it is on Hugging Face with no extra commercial terms. If you searched "run Kosmos locally", 2.5 is almost certainly what you want.

The size is counterintuitively small. The two safetensors shards in the kosmos-2.5 repo total 5.5GB stored in fp32, which works back to roughly 1.37 billion parameters; in bf16 that is about 2.8GB. What actually constrains you is not VRAM but the sequence budget. The vision side is a Pix2Struct-style variable-resolution ViT (18 layers, hidden 1536, max_num_patches 4096), and after resampling it permanently occupies 2,048 latent query tokens in the decoder — whose max_position_embeddings is only 4,096. Feed in a full page and you have barely 2,000 tokens left for output, which is exactly why the official example caps max_new_tokens at 1024. Very long table pages have to be tiled and stitched yourself.

The rest is engineering detail. Neither Kosmos2_5 nor Kosmos2 appears in vLLM's supported-architectures list, so the road is single-GPU transformers inference; and the inference.py in the unilm repo hard-depends on FlashAttention-2, which the README limits to Ampere, Ada and Hopper (3090, 4090, A100, H100) — Turing T4 and Volta V100 are out. The transformers docs hide one more trap: the `<md>` sample casts flattened_patches with a dtype variable that is never defined, so copy-pasting it raises NameError, and the height / width the processor returns must be popped before generate. These are things you find out faster by renting a card for an hour than by squinting at docs — a NexGPU RTX 3090 24GB is $0.193/GPU-hour, billed per second, cheaper than the time you would spend compiling flash-attn locally.

01 —

Kosmos checkpoints and what they cost in VRAM

Parameter counts derived from the actual weight-file sizes in each Hugging Face repo; everything here is MIT

Version	Parameters	VRAM	Context	Notes
microsoft/kosmos-2.5	1.3B (5.5GB of fp32 weights implies ~1.37B)	~5.5GB loaded as fp32 / ~2.8GB bf16; plan for 16GB+	4,096-token decoder, 2,048 permanently held by the image	The base checkpoint. Two prompts switch the task: `<ocr>` returns per-line text tagged with `<bbox>` coordinates, `<md>` returns whole-page Markdown preserving style and structure. Pre-trained on 357.4 million document pages.
microsoft/kosmos-2.5-chat	1.3B, identical architecture to the base	~2.8GB bf16, same as the base	4,096 tokens, same 2,048-token image tax	Document-VQA fine-tune driven by a "USER: {} ASSISTANT:" template — ask it for the subtotal on a receipt. The paper reports it matching models five times its size on text-rich VQA (1.3B vs 7B).
kosmos-2.5 native ckpt.pt (unilm path)	1.3B, single 6.17GB file	Pair with FlashAttention-2; a 24GB card is the safe floor for full pages	Also 4,096	The original checkpoint consumed by inference.py under microsoft/unilm/kosmos-2.5. The README states it was trained for more steps than the one reported in the paper, and that only Ampere / Ada / Hopper are supported.
microsoft/kosmos-2-patch14-224	1.6B (6.66GB of fp32 weights implies ~1.66B)	~6.7GB fp32 / ~3.3GB bf16	2,048-token decoder; the image costs only 64 latent queries	Grounding and referring, not OCR. Input is squashed to a fixed 224x224 and coordinates are quantised to a 32x32 grid (1,024 patch_index tokens), so it tells you roughly where something is — do not use it as a detector.
Kosmos-1	1.6B (per the paper)	No public weights; cannot be self-hosted	—	Paper only, with no official Hugging Face repo. Any tutorial telling you to "download Kosmos-1" is really pointing at Kosmos-2.

02 —

Which card to rent for Kosmos

NexGPU list rates on real rentable machines — mind the architecture floor FlashAttention-2 imposes

First run: pull weights, sanity-check `<md>` and `<ocr>` output formats
RTX 3090 24GB$0.193/GPU-hour
Ampere gives you native bf16 and FlashAttention-2, and 24GB swallows 2.8GB of weights plus a 4,096-patch vision forward pass with room to spare — and it is the cheapest card on the price list, a third less than a 16GB T4 while running everything the T4 cannot.
Production conversion: hundreds of thousands of PDF pages and scans into Markdown
RTX 4090 24GB$0.540/GPU-hour
Same 24GB, but Ada's compute and bandwidth chew through the 4,096-patch vision encoder far faster, and one card still has headroom for batched generation (the processor accepts a list of images).
Domain fine-tuning: receipt layouts, Chinese tables, journal typesetting
RTX A6000 48GB$0.817/GPU-hour
Full fine-tuning 1.3B with AdamW puts weights, gradients and optimiser state past 20GB before activations from a 4,096-patch vision tower and a 4,096-long sequence land on top. 48GB is where you stop sprinkling gradient checkpointing everywhere. Want it faster? A100 PCIE 80GB is $0.824/GPU-hour — under a cent more.
Large-scale pipeline: million-page archives, many processes feeding images
A100 SXM4 80GB$1.088/GPU-hour
80GB fits several model replicas side by side to saturate the SMs, and NexGPU nodes take up to 14 GPUs, so IO-bound PDF splitting and GPU inference decouple cleanly.

03 —

Four steps to a running Kosmos-2.5

Native transformers path, no flash-attn compile required; the unilm script route is covered in step four

01
Start an instance and pull the weights
Spin up an RTX 3090 24GB at console.nexgpu.net on a PyTorch prebuilt image. The repo is roughly 11.7GB in total: 5.5GB of safetensors shards plus a 6.17GB native ckpt.pt — skip the latter if you are staying on the transformers path.
```
hf download microsoft/kosmos-2.5 --local-dir ./kosmos-2.5 --exclude "ckpt.pt"
```
02
Install dependencies and confirm the Kosmos2_5 class exists
Kosmos2_5ForConditionalGeneration only ships in recent transformers releases. Import it once right after install so you find out about a stale version now rather than halfway through a batch.
```
pip install -U transformers accelerate pillow && python -c "from transformers import Kosmos2_5ForConditionalGeneration; print('ok')"
```
03
Whole-page Markdown: defuse the height/width and dtype traps
The processor returns extra height and width entries (the pre-resize dimensions) that must be popped before generate or the call errors out, and flattened_patches has to be cast to the model dtype by hand — the dtype variable in the official docs example is never defined anywhere.
```
inputs = processor(text="<md>", images=image, return_tensors="pt").to(model.device); h, w = inputs.pop("height"), inputs.pop("width"); inputs["flattened_patches"] = inputs["flattened_patches"].to(torch.bfloat16)
```
04
Need coordinates? Switch to `<ocr>` and rescale boxes to the original image
`<ocr>` emits `<bbox><x_12><y_34><x_56><y_78></bbox>` followed by that line's text. Those coordinates live on the resized canvas, so multiply them back by raw_height/height and raw_width/width. For extreme aspect ratios — long till receipts — turn on the preprocessing flags first; the accuracy difference is obvious.
```
python inference.py --do_ocr --image page.png --ckpt ckpt.pt --use_preprocess --hw_ratio_adj_upper_span "[1.5, 5]"
```

What one real batch conversion actually costs

Price out 100,000 scanned pages into Markdown. Start on an RTX 3090 24GB for two hours to tune prompts and validate output format: $0.193 x 2 = $0.386. Then move the batch to an RTX 4090 24GB and say it occupies 40 hours: $0.540 x 40 = $21.60. Storage at 50GB (5.5GB of weights plus source images plus output) at the $0.414/GB-month median works out to 40 / 730 = 0.0548 of a month, so 50 x 0.414 x 0.0548 = $1.13. Pulling roughly 8GB of Markdown back out at the $0.0081/GB median egress costs 8 x 0.0081 = $0.065. Total: 0.386 + 21.60 + 1.13 + 0.065 = $23.18. No minimum, no setup fee, no quota request. Compute billing stops the instant the instance stops; only storage keeps ticking until you destroy it. That same 4090 on a monthly managed contract typically starts in the high three figures.

04 —

FAQ

What is the minimum VRAM to run Kosmos-2.5 locally — will 8GB do?

The bf16 weights are only 2.8GB, so 8GB fits in theory. The catch is that from_pretrained without an explicit dtype loads the repo's fp32 tensors, which alone are 5.5GB; add the 4,096-patch vision forward pass and the KV for 2,048 image tokens and 8GB gets tight fast. Plan for 16GB or more. A NexGPU RTX 3090 24GB is $0.193/GPU-hour, cheaper than any 16GB card on the list — there is no reason to economise on VRAM here.

Kosmos-2 or Kosmos-2.5 — which one do I want?

Two entirely different jobs. Kosmos-2 does grounding and referring: show it a photo and it answers "a snowman" with a box attached, but input is a fixed 224x224 and coordinates snap to a 32x32 grid, so small print on a document is simply invisible to it. Kosmos-2.5 uses a variable-resolution ViT that ingests up to 4,096 patches specifically to read text-intensive images, returning either `<bbox>`-tagged lines or whole-page Markdown. Documents, receipts and tables mean 2.5; general image-text grounding means 2. Both are under 3.5GB — spin up one 3090 on NexGPU and run both for well under fifty cents.

Can I serve Kosmos-2.5 with vLLM or SGLang?

Not directly. Neither Kosmos2_5 nor Kosmos2 appears in vLLM's supported-model list, so the supported routes are transformers single-GPU inference or the inference.py in the unilm repo. The upside is that a 1.3B model never needed PagedAttention-class machinery — several processes on one card behind a queue will saturate your throughput. NexGPU nodes take up to 14 GPUs, so scaling replicas sideways beats fighting a scheduler.

Why does unilm's inference.py fail immediately on my T4 or V100?

Because that code path hard-requires FlashAttention-2, and the README limits it to Ampere, Ada and Hopper (A100, 3090, 4090, H100). Turing T4 and Volta V100 are not on the list, and V100 has no native bf16 at all. Running on older silicon means switching to the transformers path in fp16. Rather than fight it, just rent Ampere: NexGPU's RTX 3090 24GB is $0.193/GPU-hour, less than the $0.298 Tesla T4 16GB.

I want to fine-tune Kosmos-2.5 on my own receipts and Chinese tables — what should I rent?

The paper pitches exactly this: adapt it to any text-intensive image task by supervised fine-tuning with different prompts, which is how kosmos-2.5-chat was made. Full fine-tuning 1.3B with AdamW starts north of 20GB for weights, gradients and optimiser state, before activations from the 4,096-patch vision tower. 48GB is the comfortable line. NexGPU lists RTX A6000 48GB at $0.817/GPU-hour and A100 PCIE 80GB at $0.824 — under a cent apart for 32GB more, so take the A100.

Is Kosmos still being updated, and is it still worth self-hosting?

Kosmos work in the unilm repo has wound down; Microsoft's newer multimodal push went to Florence-2 and the Phi multimodal line, and the last change to the two Hugging Face repos was to align with the native transformers implementation. For self-hosting, a frozen line is a feature: fixed architecture, MIT licence, permanently downloadable weights, reproducible behaviour, no API price hikes and no deprecation notices. Matching 7B-class text-rich VQA at 1.3B is a ratio nothing its size has flattened since. Spin up an RTX 3090 24GB on NexGPU at $0.193/GPU-hour, billed per second, and you will know inside half an hour whether it is the right tool.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.