Skip to main content
NexGPU

Text LLM · Medical focus

Run Baichuan's medical models on GPUs you actually control

From Baichuan2-7B at 5.1GB in int4 to Baichuan-M3-235B at 125GB in INT4, here is every variant with its real VRAM footprint, the card that fits it, and the hourly rate.

Sign up freeView pricing

Search for "Baichuan" today and you get two almost unrelated stories. One is Baichuan2-7B / 13B, open-sourced in September 2023: a general-purpose Chinese base model with a 4K context that still shows up as a baseline in Chinese LLM evaluations. The other starts in April 2025, when Baichuan Intelligence announced it was narrowing its strategy and pivoting entirely to medical AI — producing Baichuan-M1-14B, Baichuan-M2-32B, and in January 2026 the flagship Baichuan-M3-235B. Baichuan 3 and Baichuan 4 in between never shipped weights at all; they exist only as a closed API on platform.baichuan-ai.com. So the answer to "what is the latest Baichuan" is: the general-purpose open line stopped at Baichuan2, and the live open line is the medical M series.

The M series' lineage is explicit in the config files, and it dictates how you deploy. Baichuan-M2-32B declares Qwen2ForCausalLM — 64 layers, hidden size 5120, a 152,064-token vocabulary — because it is Qwen2.5-32B with medical reinforcement training on top. That means vLLM and SGLang work out of the box, provided you pass --reasoning-parser qwen3. Baichuan-M3-235B goes further: its config says Qwen3MoeForCausalLM, 94 layers, 128 experts with 8 activated per token — the Qwen3-235B-A22B MoE skeleton. (Several Chinese write-ups describe M3 as a dense model; the config file does not support that claim.) Ironically it is the from-scratch Baichuan-M1-14B that causes the most friction: BaichuanM1ForCausalLM, 20 global attention heads alongside 40 sliding-window heads with an 8192 window on odd layers, requires trust_remote_code=True, and native vLLM support is still an open issue.

VRAM leaves no room for hand-waving. Baichuan-M2-32B's bf16 weights are 67.1GB on Hugging Face, the official GPTQ-Int4 is 20.9GB, and the official Q4_K_M GGUF is 21.9GB. Baichuan-M3-235B's GPTQ-INT4 repository is 125GB, FP8 is roughly 235GB, and bf16 is roughly 470GB. NexGPU runs 1,175 verified rentable nodes and 2,498 GPUs across 75 GPU models in 51 countries and regions, up to 14 GPUs and 2,152GB of VRAM in a single node, metered per second and priced per hour with no minimum, no setup fee, and no quota request. Every tier below names the exact card and its list rate so you can check the math yourself.

01 —

Every Baichuan variant that still exists, with real VRAM

Figures come from actual Hugging Face repository weight sizes and the official README memory tables, not from estimates.

VersionParametersVRAMContextNotes
Baichuan-M3-235B (plus FP8 / GPTQ-INT4 / Q4_K_M GGUF)235B-A22B (MoE, 128 experts, 8 active)bf16 ~470GB / FP8 ~235GB / GPTQ-INT4 ~125GB40,960The January 2026 flagship: HealthBench 65.1, HealthBench Hard 44.4, Apache 2.0. The official deployment example runs TP=8 on 8×H20 96GB with EAGLE3 speculative decoding.
Baichuan-M2-32B (plus GPTQ-Int4 / Q4_K_M GGUF)32B (Qwen2.5-32B base)bf16 67.1GB / GPTQ-Int4 20.9GB / Q4_K_M GGUF 21.9GB131,072The workhorse for single-machine self-hosting: HealthBench 60.1, AIME24 83.4, Arena-Hard-v2.0 45.8, Apache 2.0, and the repo ships a draft/ folder so you can switch on EAGLE3 speculative decoding immediately.
Baichuan-M1-14B-Base / -Instruct14B (custom architecture, trained from scratch on 20T tokens)bf16 ~29GB / 4-bit ~9GB32,768The first open medical-enhanced model of its kind, with an 8192 sliding window on odd layers. Licensed under the Baichuan-M1-14B Community License rather than Apache, and needs transformers ≥ 4.47.0 with trust_remote_code=True.
Baichuan2-13B-Base / -Chat13Bbf16 27.5GB / int8 16.1GB / int4 8.6GB4,096The 13B uses ALiBi positional encoding while the 7B uses RoPE; both were trained on 2.6 trillion tokens. vLLM has a native baichuan implementation, which makes this the smoothest path for migrating an older project.
Baichuan2-7B-Base / -Chat7Bbf16 15.3GB / int8 8.0GB / int4 5.1GB4,096The lightest tier — 5.1GB in int4 — useful as a Chinese-language baseline or for running offline batch jobs on inexpensive cards.
BaichuanMed-OCR-7B / -72B7B / 72B (Qwen2.5-VL base)7B bf16 ~17GB / 72B bf16 ~145GBinherited from baseTurns lab reports and imaging scans into structured Markdown and answers questions about them; the 7B scores 83.5% on medical report benchmarks against 71.3% for stock Qwen2.5-VL-7B. Its licence field reads "other" — read the terms before shipping.

02 —

Pick your card against NexGPU list rates

KV cache math straight from config.json: M2-32B is 64 layers × 8 KV heads × 128 dims, or 256KB per token; M3-235B is 94 layers × 4 KV heads, roughly 188KB per token.

  • Baichuan-M2-32B-GPTQ-Int4 on one card, batch evaluation under 8K context

    RTX 4090 24GB$0.540/GPU-hour

    The 20.9GB of weights just fit in 24GB, leaving 2–3GB for KV — at 256KB per token that is roughly 8–10K of context, fine for short Q&A evals and an immediate OOM on a full patient record.

  • Baichuan-M2-32B INT4 at 24K–32K context for long clinical documents

    RTX 5090 32GB$0.723/GPU-hour

    After the weights, 32GB frees roughly 8GB for KV cache, which covers a 32K document — the sweet spot for single-card M2 serving.

  • Baichuan-M2-32B at full bf16 precision, or QLoRA fine-tuning

    A100 PCIE 80GB$0.824/GPU-hour

    All 67.1GB of weights fit on one card for just $0.101/hour more than the RTX 5090, which removes quantisation error entirely — at 32B, quantising saves surprisingly little money on this price list.

  • Baichuan-M3-235B-GPTQ-INT4 with four-way tensor parallelism

    A100 SXM4 80GB$1.088/GPU-hour

    125GB of weights across 4 cards means 320GB total with comfortable headroom, and SXM4 NVLink keeps the TP=4 all-reduce off the critical path — $4.352/hour for a complete 235B medical inference stack.

03 —

Four steps to a running Baichuan-M2-32B

One RTX 5090 32GB with GPTQ-Int4, from boot to first clinical query.

  1. 01

    Boot an instance with vLLM

    Pick RTX 5090 32GB at console.nexgpu.net and choose a vLLM image from the 2,000+ prebuilt images. SSH, Jupyter, web terminal, REST API, and CLI are all available. M3-235B officially requires vllm ≥ 0.9.0 or sglang ≥ 0.4.6.post1; M2-32B tracks the same line, so upgrade if your image predates it.

    pip install -U "vllm>=0.9.0" huggingface_hub
  2. 02

    Pull the official INT4 weights

    Fetch baichuan-inc/Baichuan-M2-32B-GPTQ-Int4 — four safetensors shards totalling 20.9GB. If you prefer llama.cpp, swap in baichuan-inc/Baichuan-M2-32B-Q4_K_M-GGUF, a single 21.9GB file — note it is about 1GB larger than the GPTQ build, which makes a 24GB card even tighter.

    hf download baichuan-inc/Baichuan-M2-32B-GPTQ-Int4 --local-dir /workspace/m2-int4
  3. 03

    Serve an OpenAI-compatible endpoint with context capped to fit

    The critical flag is --reasoning-parser qwen3: M2 is a reasoning model, and without it the thinking trace lands verbatim in the content field. --max-model-len 24576 corresponds to about 6GB of KV, which leaves headroom on a 32GB card; move to an RTX A6000 48GB and you can push toward 100K.

    vllm serve /workspace/m2-int4 --served-model-name Baichuan-M2-32B --reasoning-parser qwen3 --max-model-len 24576 --gpu-memory-utilization 0.92
  4. 04

    Send a real clinical query, then stop billing by the second

    Baichuan recommends temperature 0.6, with max_tokens at 32k for evaluation runs and 64k for AIME-style tasks. Stop the instance when you are done and compute billing stops immediately; the data volume keeps billing at the $0.414/GB-month median until you destroy it.

    curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"Baichuan-M2-32B","temperature":0.6,"messages":[{"role":"user","content":"58-year-old male, chest pain for 2 hours, ST elevation in leads II, III, aVF. What is the next step?"}]}'

What one real evaluation run actually costs

Say you run 5,000 internal clinical Q&A items through Baichuan-M2-32B-GPTQ-Int4 on a single RTX 5090 32GB. Boot, pulling 20.9GB of weights, and warm-up take about 0.4 hours; the evaluation itself takes 5.6 hours, for 6 hours total: 6 × $0.723 = $4.34. Weights plus the Python environment sit on a 60GB volume you keep for 3 days in case you re-run: 60 × $0.414 × 3/30 = $2.48. Exporting 2GB of results: 2 × $0.0081 = $0.02. Total, about $6.84. The interesting comparison is the control: the same 6 hours at full bf16 precision on an A100 PCIE 80GB is 6 × $0.824 = $4.94 — only $0.60 more than the 5090 tier, with the entire quantisation step removed. At 32B, quantising is about fitting into a small card, not about saving money. One tier up, Baichuan-M3-235B-GPTQ-INT4 on 4×A100 SXM4 80GB is 4 × $1.088 = $4.352/hour, so an 8-hour department-scale load test costs $34.82; the same 8 hours at native FP8 on 2×H200 141GB is 2 × $6.660 × 8 = $106.56. The gap between those tiers is real — whether it is worth it depends on whether you need a benchmark number or production-grade precision.

04 —

FAQ

Is Baichuan still being updated, and which version should I use in 2026?

The general-purpose open line stopped at Baichuan2-7B / 13B in September 2023, and Baichuan 3 and 4 never released weights. In April 2025 Baichuan Intelligence narrowed its strategy to medical AI, and everything open-sourced since has been medical: Baichuan-M1-14B, Baichuan-M2-32B, and Baichuan-M3-235B in January 2026. For general Chinese tasks Baichuan2 still works but has clearly fallen behind; for medical Q&A, clinical record understanding, or decision support, go straight to M2 or M3. Any of these can be spun up on NexGPU on a card with matching VRAM, billed per second, and stopped the moment you are done.

Can Baichuan-M2-32B really run on a single RTX 4090?

Yes, but know where the wall is. The official GPTQ-Int4 weights are 20.9GB, so a 24GB card has 2–3GB left. M2's KV cache costs 256KB per token (64 layers × 8 KV heads × 128 dims × 2 bytes × K and V), which works out to roughly 8–10K of context: plenty for short Q&A evaluation, an instant OOM on a complete inpatient record. For 32K move to an RTX 5090 32GB at $0.723/GPU-hour; to approach the full 128K window take an RTX A6000 48GB at $0.817/GPU-hour. All three are in stock on NexGPU, and swapping cards is a few minutes of work.

How much VRAM and how many GPUs does self-hosting Baichuan-M3-235B need?

It depends on precision. bf16 weights run about 470GB, which needs 4×H200 141GB (564GB, $26.64/hour) or 8×H100 SXM 80GB. The FP8 build is about 235GB, so 2×H200 141GB gives you 282GB and fits neatly at $13.32/hour — note FP8 needs Hopper silicon for native throughput. The GPTQ-INT4 repository measures 125GB, and 4×A100 SXM4 80GB for 320GB is the best value at $4.352/hour. Its KV cache is cheap: 94 layers with only 4 KV heads is about 188KB per token, so even the full 40K window costs just 7.3GB. NexGPU nodes go up to 14 GPUs and 2,152GB of VRAM, so every one of these layouts fits inside a single node — no cross-node fabric required.

How do I fix "'BaichuanTokenizer' object has no attribute 'sp_model'" when loading Baichuan2?

That is a breaking change introduced in transformers 4.34 colliding with Baichuan2's bundled remote-code tokenizer; it was debated at length in the project's issues. The quickest fix is to pass revision="v2.0" when loading, or pin transformers to 4.33.x. The other route is to skip transformers entirely and use vLLM's native baichuan implementation, which does not depend on that remote code. Also note Baichuan2 needs PyTorch 2.0 or newer to use F.scaled_dot_product_attention, and CPU inference must switch to float32. On NexGPU's PyTorch images you can build an isolated environment and pin those versions without fighting your local machine.

Are all Baichuan versions free for commercial use? Do they share a licence?

They do not, and this trips people up. Baichuan-M2-32B and Baichuan-M3-235B are Apache 2.0, open to both research and commercial use. Baichuan-M1-14B ships under the Baichuan-M1-14B Community License. Baichuan2 combines Apache 2.0 with a community licence agreement: commercial use requires emailing [email protected] and meeting three conditions — under 1 million daily active users, not being a software or cloud service provider, and no sub-licensing to third parties. The BaichuanMed-OCR models list their licence simply as "other," so read the terms before you ship. Every M-series model also carries a medical disclaimer: research and reference only, never a substitute for professional diagnosis or treatment.

How many GPUs do I need to fine-tune Baichuan-M2-32B?

Full-parameter bf16 fine-tuning has three line items: 65GB of weights, 65GB of gradients, and roughly 394GB for AdamW's fp32 master weights plus first and second moments at 12 bytes per parameter — over 500GB in total, which in practice means 8×A100 80GB (640GB) with ZeRO-3, or 8 × $1.088 = $8.704/hour on NexGPU. Most medical use cases never need that. QLoRA drops the base model to 4-bit, and a single A100 PCIE 80GB at $0.824/GPU-hour carries LoRA training on 32B comfortably; at short sequence lengths even an RTX A6000 48GB at $0.817/GPU-hour will do. Get your data pipeline and hyperparameters working on one card first, then scale out — per-second billing means that kind of experimentation carries almost no sunk cost, and bilingual support is on Telegram with no ticket queue when you hit something odd.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.