On-device LLM

Run MiniCPM yourself, from 4GB of VRAM to a million tokens

The MiniCPM family from ModelBest and Tsinghua's THUNLP spans 1.08B MiniCPM5-1B up to the 9B omnimodal MiniCPM-o 4.5 — all Apache-2.0, all free to use commercially. NexGPU starts at $0.193/GPU-hour, cheap enough to benchmark the entire line in one evening.

MiniCPM · self-hosted

MiniCPM is close to a synonym for "on-device LLM" in the Chinese open-source scene. It is maintained by ModelBest together with Tsinghua University's Natural Language Processing Lab (THUNLP), with code and weights published under the OpenBMB org: language models at github.com/OpenBMB/MiniCPM, the multimodal MiniCPM-V and omnimodal MiniCPM-o at github.com/OpenBMB/MiniCPM-V, and a separate deployment cookbook at OpenSQZ/MiniCPM-V-CookBook. The whole line is Apache-2.0 — weights and code, commercial use included, with no separate agreement to sign.

If your mental model of MiniCPM is the 2.4B MiniCPM-2B from 2024, you are two generations behind. The current line-up: MiniCPM4.1-8B (September 2025) with InfLLM v2 trainable sparse attention and a hybrid reasoning switch; MiniCPM-SALA (February 2026), a sparse-and-linear hybrid that scales past 1M tokens; MiniCPM-o 4.5 (February 2026), end-to-end omnimodal with full-duplex streaming speech; MiniCPM-V 4.6 (May 2026), multimodal compressed down to 1.3B so it fits on a phone; and MiniCPM5-1B (May 2026), the newest on-device text workhorse. Any tutorial still walking you through MiniCPM-2B-dpo can be skipped.

MiniCPM earns its deployment economics by being small — but small does not mean any card will do. MiniCPM-V 4.6 at 1.3B is officially 4GB of GPU memory, 2–3GB once quantised. MiniCPM-o 4.5 at 9B needs 19.0GB in bf16 before you leave room for video frames and the audio encoder. The tables below use figures published by OpenBMB plus weight sizes you can derive directly from parameter count and dtype, then match NexGPU cards against them honestly — no guessing, no recycling 70B advice onto an 8B model.

01 —

Current MiniCPM variants and what they actually cost in VRAM

Parameters, context, and quantised footprints taken from the official OpenBMB model cards and repo READMEs

Version	Parameters	VRAM	Context	Notes
MiniCPM5-1B	1.08B total (0.68B non-embedding)	bf16 weights ≈ 2.2GB / Q4-class GGUF ≈ 0.8GB	131,072 (128K)	Released May 2026 as the newest on-device text model. LlamaForCausalLM architecture, 24 layers, GQA with 16 query heads to 2 KV heads, and a built-in <think> chat template gated by enable_thinking. OpenBMB reports that RL plus On-Policy Distillation in post-training lifted the average score by 16 points while cutting overlong responses by 29 percentage points. SGLang is the recommended backend for tool calling.
MiniCPM4.1-8B	8B	bf16 weights ≈ 16GB / GPTQ-Int4 ≈ 9GB / GGUF Q4_K_M 4.97GB	65,536 native; 131,072 validated via LongRoPE	The practical workhorse. InfLLM v2 trainable sparse attention with a dense-sparse switchable path, plus hybrid reasoning you toggle with enable_thinking=True/False in the tokenizer's chat template. OpenBMB measured 3x faster decoding in reasoning mode on an RTX 4090. Official quantised releases cover AutoAWQ, GPTQ, Marlin, MLX and GGUF.
MiniCPM-SALA	9B	bf16 weights ≈ 18GB, plus KV cache for whatever context you actually use	1M+ tokens	The long-context specialist from February 2026. Sparse-And-Linear-Attention: 25% InfLLM-V2 sparse layers and 75% Lightning Attention linear layers, with Hybrid Positional Embedding (HyPE). OpenBMB measured 3.5x the inference speed of Qwen3-8B at 256K sequences on an A6000D, and notes Qwen3-8B goes OOM at 1M tokens where SALA completes. Recommended temperature is 0.9.
MiniCPM-V 4.6	1.3B (SigLIP2-400M vision tower + Qwen3.5-0.8B LLM)	bf16 ≈ 4GB / int4 (BNB, AWQ, GPTQ) ≈ 3GB / GGUF ≈ 2GB	256K	The smallest MiniCPM-V ever shipped, merged into Ollama's official library in June 2026. Mixed 4x/16x visual token compression cuts visual encoding FLOPs by more than half while still handling single-image, multi-image and video. Video defaults to 1 FPS for clips up to 128 seconds, capped at 128 frames. Note that a separate MiniCPM-V-4.6-Thinking checkpoint exists — do not confuse it with the Instruct weights.
MiniCPM-o 4.5	9B (SigLip2 + Whisper-medium + CosyVoice2 + Qwen3-8B)	bf16 19.0GB / int4 11.0GB (officially measured)	Images up to 1.8M pixels; video up to 10fps at any aspect ratio	End-to-end omnimodal: vision, audio and text in, text and speech out, with full-duplex streaming and 1Hz proactive decision-making. OpenCompass 77.6. The official efficiency table lists 154.3 tok/s decoding at bf16 and 212.3 tok/s at int4, both with 0.6s TTFT. Known limitations include occasionally unstable speech output and mixed English-Chinese responses in omni mode.
MiniCPM4-8B / MiniCPM4-0.5B / BitCPM4	8B / 0.5B / 1B ternary	MiniCPM4-0.5B-QAT-Int4 weights ≈ 0.3GB	32K on MiniCPM4-8B	The June 2025 generation, still maintained in the repo. MiniCPM4-0.5B-QAT-Int4 uses quantisation-aware training and BitCPM4-1B/0.5B are the ternary line — pick these only when you are genuinely shipping to edge silicon such as Jetson AGX Orin. For server-side self-hosting, go straight to 4.1-8B.

02 —

Which GPU to rent for which MiniCPM

Every NexGPU rate below is public list pricing, metered per second and billed per hour, with no minimum and no setup fee

MiniCPM5-1B or MiniCPM-V 4.6 for prompt regression and quality checks
RTX 3090 24GB$0.193/GPU-hour
MiniCPM-V 4.6 needs only 4GB in bf16, so 24GB swallows the model plus a 256K KV cache with room to spare — and the 3090 is the cheapest card on our list with full bf16 support.
Serving quantised MiniCPM4.1-8B (GPTQ-Int4 or GGUF Q4_K_M) through vLLM under real concurrency
RTX 4090 24GB$0.540/GPU-hour
The official 3x reasoning-mode decode speedup was measured on an RTX 4090. Nine gigabytes of Int4 weights leaves roughly 15GB for KV cache and batching — enough for a genuine internal API on one card.
MiniCPM4.1-8B at full bf16 with 128K LongRoPE, or MiniCPM-o 4.5 omnimodal in bf16
RTX 5090 32GB$0.723/GPU-hour
MiniCPM-o 4.5 measures 19.0GB in bf16; 32GB fits it with headroom for video frames and the audio encoder. The 8B model's 16GB of bf16 weights plus long-context KV also lands squarely in this tier — 24GB gets tight.
MiniCPM-SALA at 1M tokens, or benchmarking several MiniCPM variants side by side
RTX A6000 48GB$0.817/GPU-hour
OpenBMB's 3.5x SALA speedup number was measured on an A6000D. With 48GB you keep 9B weights, a million-token KV cache and Lightning Attention's intermediate state resident at once, with no tensor parallelism to tune.

03 —

Four steps to a running MiniCPM

NexGPU offers 2,000+ prebuilt images including PyTorch and vLLM, reachable over SSH, Jupyter, web terminal, REST API or CLI

01
Boot the box and pin your versions first
Pick a card at console.nexgpu.net, attach a vLLM or PyTorch image, and check framework versions before anything else. MiniCPM leans hard on recent architecture support: MiniCPM-V 4.6 requires vLLM ≥ 0.22.0 and transformers ≥ 5.7.0, and an older stack simply reports an unrecognised architecture. Historically vLLM 0.6.x hit both a min_thread_n divisibility error and an absmax KeyError on MiniCPM quantised weights, so err on the newer side.
```
nvidia-smi && python -c "import torch, transformers, vllm; print(torch.__version__, transformers.__version__, vllm.__version__)"
```
02
Pull the weights
Everything lives under the openbmb org on Hugging Face. Grab MiniCPM4.1-8B for general text, MiniCPM5-1B for the on-device tier, MiniCPM-V-4.6 for vision, MiniCPM-o-4_5 for omnimodal. If VRAM is tight, pull the official quantised repos directly: MiniCPM4.1-8B-GPTQ, MiniCPM4.1-8B-AutoAWQ, or MiniCPM4.1-8B-GGUF, whose Q4_K_M file is 4.97GB. NexGPU storage runs a $0.414/GB-month median; compute billing stops the moment the instance stops, but storage keeps accruing until you destroy it.
```
hf download openbmb/MiniCPM4.1-8B --local-dir ./MiniCPM4.1-8B
```
03
Serve it behind an OpenAI-compatible endpoint
Every MiniCPM needs --trust-remote-code; the official modeling files require it and the server will not start without it. Bring it up at the native 65,536 context first, confirm your throughput and memory curve, and only then edit rope_scaling in config.json to reach 128K via LongRoPE — extending context is a config change, not a launch flag, and that is where most people stall. Deep thinking is controlled per request through enable_thinking in chat_template_kwargs.
```
vllm serve openbmb/MiniCPM4.1-8B --trust-remote-code --max-model-len 65536 --port 8000
```
04
Branch out: multimodal, sparse attention, GGUF
For vision, start MiniCPM-V-4.6 with a modest --max-model-len 8192 and climb toward 256K once it is stable. If you see runaway generation, the official recipe prescribes stop_token_ids: [248044, 248046] — a consequence of the new Qwen3.5 vocabulary. Video input needs the vllm[video] extra, and a torchcodec/CUDA mismatch throws "Could not load libtorchcodec", fixed by aligning torch with the CUDA in your image. For the real InfLLM v2 sparse-attention speedup, note that vLLM currently takes the dense path — use Transformers or OpenBMB's CPM.cu instead. For CPU or small-VRAM boxes, run the GGUF under llama.cpp.
```
./llama-cli -m MiniCPM4.1-8B-Q4_K_M.gguf -p "What is worth visiting in Beijing?" -n 1500
```

What a full MiniCPM evaluation actually costs

Say you want to work through the whole line in two days and pick the right tier for your product. Start on an RTX 3090 24GB at $0.193/GPU-hour with MiniCPM5-1B and MiniCPM-V 4.6, three hours to run your prompt regression suite: 3 × $0.193 = $0.579. Move to an RTX 5090 32GB at $0.723/GPU-hour for MiniCPM4.1-8B in bf16 and MiniCPM-o 4.5 in bf16 (19.0GB measured, comfortably inside 32GB), five hours of load testing: 5 × $0.723 = $3.615. Storage: roughly 16GB for 4.1-8B bf16, 18GB for o 4.5, and 6GB for the two small models — 40GB held for two days at the $0.414/GB-month median works out to 40 × 0.414 × 2 ÷ 30 ≈ $1.10. Pull 2GB of logs and results back down at the $0.0081/GB median egress: 2 × 0.0081 ≈ $0.02. Total: $0.579 + $3.615 + $1.10 + $0.02 ≈ $5.31. Under six dollars for a conclusion drawn on your own data instead of somebody else's leaderboard. Compute is metered per second and stops when the instance stops; storage runs until you destroy it, so wipe the volume when the evaluation is done.

04 —

FAQ

How much VRAM do I actually need to run MiniCPM locally?

It depends entirely on which one. MiniCPM-V 4.6 (1.3B) is officially about 4GB in bf16, roughly 3GB at int4, and around 2GB as GGUF. MiniCPM5-1B's 1.08B weights come to about 2.2GB in bf16. MiniCPM4.1-8B is roughly 16GB in bf16, about 9GB as GPTQ-Int4, and the official Q4_K_M GGUF file is 4.97GB. MiniCPM-o 4.5 measures 19.0GB bf16 and 11.0GB int4. Add KV cache and framework overhead and the picture is simple: the 1B tier is comfortable on an RTX 3090 24GB at $0.193/GPU-hour, quantised 8B belongs on an RTX 4090 24GB at $0.540/GPU-hour, and 9B omnimodal in bf16 wants an RTX 5090 32GB at $0.723/GPU-hour. All three are in stock on NexGPU, billed by the second, stopped the moment you are done.

How do I really get 128K context out of MiniCPM4.1-8B?

The native context is 65,536; 131,072 requires configuring LongRoPE rope_scaling in config.json, an extension OpenBMB has validated on long-context evaluations. It is not a launch flag, and once you enable it you have to re-check KV cache memory — a 128K KV cache comfortably exceeds the weights themselves, and that is exactly where 24GB cards go OOM. Either step up to an RTX 5090 32GB at $0.723/GPU-hour, or switch to MiniCPM-SALA, which was architected for million-token contexts, on an RTX A6000 48GB at $0.817/GPU-hour. On NexGPU, changing cards means launching a different instance — no quota request, no waiting.

I'm running MiniCPM4.1-8B on vLLM and I don't see the sparse-attention speedup. Why?

Wrong path. The official docs are explicit: the dense attention route runs on vLLM, SGLang and Transformers, while InfLLM v2 sparse attention is currently supported through Hugging Face Transformers and OpenBMB's own CPM.cu, which they "highly recommend" for maximum speed. The 3x reasoning-mode decode figure on RTX 4090 was measured on that path, not on vLLM. To verify it you need a separate CPM.cu install and a head-to-head run. Two hours on an RTX 4090 24GB at $0.540/GPU-hour costs $1.08 on NexGPU — considerably less painful than fighting a local CUDA environment.

Can I use MiniCPM commercially? What is the licence?

Yes. Both the language-model repo and the MiniCPM-V / MiniCPM-o multimodal repo release weights and code under Apache-2.0, and the official README states plainly that model weights and code are open-sourced under Apache-2.0 with commercial use permitted. No separate agreement, no per-seat authorisation. The project is maintained jointly by ModelBest and Tsinghua University's THUNLP, hosted under the OpenBMB org. That leaves you needing only a machine to run it on — and NexGPU spans 51 countries and regions with 1,175 verified rentable nodes and 2,498 GPUs, so you choose the region your data sits in.

MiniCPM-V 4.6 is only 1.3B — why won't my deployment stop generating?

That is the new vocabulary that came with the Qwen3.5 backbone in v4.6. The official vLLM recipe gives the fix directly: if you see runaway generations, pass stop_token_ids: [248044, 248046]. Also confirm vLLM ≥ 0.22.0 and transformers ≥ 5.7.0, since older versions cannot even resolve the architecture. One more common trap: v4.6 shipped as two independent checkpoints, Instruct and Thinking, and mixing them up changes the output behaviour completely. For version-sensitive stacks like this, the cleanest fix is a fresh instance built from scratch — NexGPU has 2,000+ prebuilt images with PyTorch and vLLM ready at boot, and bilingual support over Telegram with no ticket queue.

At 8B–9B, should I pick MiniCPM or something else?

MiniCPM differentiates on efficiency rather than raw ceiling. MiniCPM-o 4.5 reaches OpenCompass 77.6 at 9B while occupying 19.0GB in bf16, where comparable omnimodal models go straight to OOM under the same conditions. MiniCPM-SALA runs 3.5x faster than Qwen3-8B at 256K sequences and completes at 1M tokens where Qwen3-8B does not. MiniCPM-V 4.6 covers single-image, multi-image and video at 1.3B. If your binding constraint is VRAM budget, edge deployment, or long-context throughput, MiniCPM is usually the better economics. The only reliable way to decide is to run your own data through it — NexGPU offers 75 GPU models from a $0.188/GPU-hour Tesla V100 32GB to a $6.660/GPU-hour H200 141GB, up to 14 GPUs per node and 2,152GB max node VRAM, metered per second and stopped when you stop.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.