Text LLM

Self-hosting Yi: the VRAM math for 6B through 34B

01.AI's open platform stops serving API calls on 3 September 2026, which makes the Apache 2.0 Yi weights the only way to keep using these models. Here is the real memory footprint of every Yi-1.5 and Yi-Coder variant, and the card to rent for each.

Yi · self-hosted

Yi is the model family 01.AI trained from scratch. The first generation landed in November 2023 on 3.1T multilingual tokens, shipping 6B, 9B and 34B at once plus separate 200K-context variants. Yi-1.5, released May 2024, continued pre-training on top of that with another 500B tokens (3.6T cumulative) and 3M fine-tuning samples, lifting coding, math and instruction following a full tier. Yi-Coder followed in September 2024, adding 2.4T code tokens across 52 programming languages on top of Yi-9B and opening the window to 128K. That is the whole open-weight line: the 01-ai repositories on Hugging Face were last touched in November 2024, after which the company moved its focus to its Wanzhi enterprise platform.

So most people searching for how to self-host Yi today are not chasing the frontier — they have a deadline. On 10 August 2026 01.AI announced that its open platform will wind down the web playground, API access and account top-ups; API calls stop at 24:00 on 3 September 2026, and the refund window closes on 3 December. The closed-weight Yi-Lightning MoE goes dark with it. Yi-1.5 and Yi-Coder, however, are Apache 2.0 — anyone can pull them and nobody can take them back. Moving those calls onto a machine you control is the only continuation path, and it happens to be the low-drama one.

The good news is that Yi deploys with essentially zero adaptation work: the architectures field in config.json is plain LlamaForCausalLM, so vLLM, SGLang, TGI and llama.cpp all support it natively with no trust_remote_code. Three traps are worth knowing up front. The vocabulary is 64,000 tokens, not Llama's 32,000, and the chat format is ChatML — <|im_start|> / <|im_end|>, not [INST] — so the wrong template produces pure garbage. rope_theta is 5,000,000 (10,000,000 for Yi-Coder), not the usual 10000. And most painfully, Yi-1.5-34B-Chat, 9B-Chat and 6B-Chat all pin max_position_embeddings at 4096, which vLLM enforces as a hard ceiling; getting a longer window means switching repository, not switching flags. Every number below respects those constraints.

01 —

Yi variants and what they actually cost in VRAM

Parameter counts, quantised file sizes and the real usable window, taken from the official configs and published GGUF builds

Version	Parameters	VRAM	Context	Notes
Yi-1.5-34B-Chat-16K	34.4B (60 layers / 56 heads / 8 KV heads, GQA)	bf16 ~69GB; Q8_0 36.54GB; Q6_K 28.21GB; Q5_K_M 24.32GB; Q4_K_M 20.65GB; IQ4_XS 18.47GB	16K (max_position_embeddings=16384)	The strongest thing in the open Yi line and the only 34B chat checkpoint whose config genuinely declares a long window. Use this repo for RAG, long-document summarisation and multi-turn support; do not try to stretch Yi-1.5-34B-Chat instead. Eight KV heads keep the cache at 240KiB per token, so a full 16K sequence costs about 3.75GB.
Yi-1.5-34B-Chat	34.4B	bf16 ~69GB; Q5_K_M 24.32GB; Q4_K_M 20.65GB; Q3_K_M 16.65GB	4K (max_position_embeddings=4096)	Same training lineage as the 16K build, scoring 52.29 on MMLU-Pro, with 01.AI describing it as matching or beating larger models on most benchmarks. But the window is nailed to 4096 and vLLM will cap you there. Pick it for short Q&A, classification and field extraction, where the small KV footprint buys you the highest concurrency.
Yi-1.5-9B-Chat-16K	8.83B (48 layers / 32 heads / 4 KV heads)	bf16 ~17.7GB; Q8_0 9.38GB; Q6_K 7.24GB; Q5_K_M 6.25GB; Q4_K_M 5.32GB	16K	The single-card sweet spot, and 01.AI's pick for best open model at its size. With only 4 KV heads the cache is 96KiB per token, so a full 16K sequence is about 1.5GB — a 24GB card holds the bf16 weights and still has room for a dozen concurrent streams, which is rare at this size.
Yi-1.5-6B-Chat	6.06B	bf16 ~12.1GB (01.AI lists 15GB minimum); Q8_0 6.44GB; Q5_K_M 4.30GB; Q4_K_M 3.67GB	4K	The lowest bar in the family. At 3.67GB for Q4_K_M it fits a 16GB Tesla T4 with room to spare. Good as a pipeline worker for intent classification, rewriting and data cleaning, where you want several replicas side by side on one machine.
Yi-Coder-9B-Chat	8.83B (Yi-9B plus 2.4T additional code tokens across 52 languages)	bf16 ~17.7GB; Q8_0 9.38GB; Q6_K 7.25GB; Q4_K_M 5.33GB; a full 128K sequence adds roughly 12GB of KV cache	128K (max_position_embeddings=131072, rope_theta=1e7)	85.4% on HumanEval, 73.8% on MBPP, 23.4% on LiveCodeBench, and 70.3% average across seven program-aided math benchmarks — ahead of the three-times-larger DeepSeek-Coder-33B. Genuinely the sub-10B model that can hold a repository in context. A 1.5B sibling exists for autocomplete.
Yi-34B-200K (first-generation Yi)	34.4B	01.AI lists 200GB minimum; broken down that is 69GB of bf16 weights plus roughly 48GB of KV per full 200K sequence	200K	Yi-1.5 has no 200K variant at all — it tops out at 32K via the Yi-1.5-34B-32K and Yi-1.5-9B-32K base models. A genuine 200,000-token window means going back to the first-generation repos, which also include Yi-6B-200K (50GB minimum) and Yi-9B-200K.

02 —

Which card to rent

Matched honestly: fitting the weights is only the pass mark, the headroom left for KV cache decides how long a context you can actually serve

Single card running Yi-1.5-9B-Chat-16K or Yi-1.5-6B-Chat at native bf16
RTX 3090 24GB$0.193/GPU-hr
The 9B is 17.7GB in bf16 and the 6B only 12.1GB, so 24GB leaves real KV headroom — and the 3090 costs less than the 16GB Tesla T4 at $0.298/GPU-hr, which makes it hard to justify anything else in this tier.
Single card running Yi-1.5-34B-Chat as Q4_K_M GGUF or the official AWQ 4-bit build
RTX 5090 32GB$0.723/GPU-hr
Q4_K_M weights are 20.65GB. On a 24GB card, once fragmentation is accounted for you have two or three GB for KV cache while a full 16K window needs 3.75GB — it will OOM. 32GB leaves roughly 10GB, enough for 16K plus several concurrent requests.
Yi-1.5-34B-Chat-16K at bf16 for formal evaluation or a production endpoint
A100 PCIE 80GB$0.824/GPU-hr
Keeping all 69GB of weights on one card avoids tensor-parallel communication overhead entirely, and the remaining ~10GB covers KV for a 16K window. Step up to A100 SXM4 80GB ($1.088/GPU-hr) or H100 SXM 80GB ($3.582/GPU-hr) when you need the throughput.
Yi-Coder-9B-Chat with the full 128K window open across a whole repository
RTX A6000 48GB$0.817/GPU-hr
17.7GB of weights plus roughly 12GB of KV for one full-length 128K sequence; 48GB lets you serve two or three long-context requests at once. A 24GB card realistically caps out near 32K, which is not enough for a mid-sized repo.

03 —

Four steps to a running Yi endpoint

From boot to an OpenAI-compatible API in under twenty minutes; the commands are copy-paste ready

01
Launch an instance with vLLM already on it
Pick a card from the table above in the NexGPU console and choose one of the prebuilt vLLM or PyTorch images. CUDA, PyTorch and vLLM are already installed across 2,000+ images, so there is no environment to assemble and no quota request to file. Get in over SSH, Jupyter or the web terminal — whichever you prefer. Billing starts the second the instance is actually running and is metered per second.
02
Pull the weights onto the instance
The whole family lives under the 01-ai organisation on Hugging Face under Apache 2.0 — no access request, no agreement to sign, no regional gate. Nodes in China usually pull faster from the mirrored 01ai repositories on ModelScope. The 34B's 69GB of bf16 weights take roughly ten minutes on a gigabit link; if you just want to sanity-check the family first, grab the 9B and be running in two or three.
```
pip install -U huggingface_hub && hf download 01-ai/Yi-1.5-34B-Chat-16K --local-dir /workspace/Yi-1.5-34B-Chat-16K
```
03
Serve an OpenAI-compatible endpoint with vLLM
Yi's architectures field is LlamaForCausalLM, so vLLM loads it natively with no trust_remote_code. The one flag that matters is --max-model-len: Yi-1.5-34B-Chat declares 4096 and only the -Chat-16K repository declares 16384, and vLLM enforces whatever the config says. Forcing a value beyond the trained length just yields noise. Once it is up, point the base_url in your existing 01.AI API code at this endpoint and the rest of the integration stays as it is.
```
vllm serve /workspace/Yi-1.5-34B-Chat-16K --served-model-name yi-34b --dtype bfloat16 --max-model-len 16384 --gpu-memory-utilization 0.92 --port 8000
```
04
Drop to GGUF when VRAM is the binding constraint
On a 24-32GB card, run Q4_K_M through llama.cpp instead: the 34B shrinks to 20.65GB and -ngl 99 pushes all 60 layers onto the GPU, with quality loss that is hard to notice in Chinese conversation. Remember the ChatML template — <|im_start|>system, <|im_start|>user, <|im_start|>assistant, each closed with <|im_end|> — and the 64,000-token vocabulary. Feeding Yi a Llama-style [INST] prompt produces immediate gibberish.
```
llama-server -m /workspace/Yi-1.5-34B-Chat-Q4_K_M.gguf -ngl 99 -c 16384 --host 0.0.0.0 --port 8000
```

What one evaluation run actually costs

Take a real job. On an A100 PCIE 80GB at $0.824/GPU-hr, serving Yi-1.5-34B-Chat-16K in bf16 to run 5,000 internal evaluation prompts: about 3 minutes to boot the prebuilt vLLM image, about 10 minutes to pull 69GB of weights over a gigabit link, about 4 minutes to load and warm up, and roughly 2.5 hours for the evaluation itself — call it 2.8 hours. Compute: 2.8 x $0.824 = $2.31. Keep the weights on disk for a week for the next iteration: 69GB x $0.414 / 30 days x 7 days = $6.67. Export 1.2GB of results: 1.2 x $0.0081 = $0.01. Total $8.99, of which only $2.31 is actual GPU time. Run the same evaluation at Q4_K_M (20.65GB) on an RTX 5090 32GB at $0.723/GPU-hr: throughput is a little lower, so about 3.4 hours x $0.723 = $2.46, and the smaller weights cost 20.65 x $0.414 / 30 x 7 = $2.00 to keep for the week. That is roughly $4.46, less than half the bf16 route. If you are only scoping out the 9B first, an RTX 3090 24GB is $0.193/GPU-hr — the price of a coffee buys you fifty-odd hours. The line most people forget is storage. Stop the instance and compute billing stops that second, but the weights on disk keep billing until you destroy the volume. Stop the instance when the run finishes and leave the weights in place so the next boot is instant; delete the volume once you are done iterating. No minimum spend, no setup fee, no quota approval — the three hidden costs of a conventional cloud are all zero here.

04 —

FAQ

01.AI is shutting down its API. Can I still use Yi?

The open platform stops serving the playground and API at 24:00 on 3 September 2026, new registrations and top-ups are already closed, and the refund window runs to 24:00 on 3 December. The closed-weight Yi-Lightning goes with it. Yi-1.5 and Yi-Coder, though, are Apache 2.0 — anyone can download them and nobody can revoke them, so standing up your own endpoint is both the only and the most durable continuation. Point base_url at your own vLLM instance and your application code barely changes. NexGPU bills per second with no minimum, so most teams find the GPU hours cost less than the API invoice did.

How much VRAM does Yi-1.5-34B need to run locally?

In bf16 the weights are 34.4B x 2 bytes, about 69GB, and 01.AI lists 72GB as the practical minimum, so a single card means the 80GB class. Q4_K_M lands at 20.65GB, the official AWQ 4-bit build is listed at 20GB, and the GPTQ 8-bit build at 38GB. Mapped onto NexGPU: bf16 on an A100 PCIE 80GB at $0.824/GPU-hr, 4-bit on an RTX 5090 32GB at $0.723/GPU-hr, 8-bit on an RTX A6000 48GB at $0.817/GPU-hr. Budget for KV cache on top of the weights — Yi-1.5-34B burns 240KiB per token.

Can a single RTX 4090 24GB run Yi-34B?

It will start, but it is tight. Q4_K_M is 20.65GB and after fragmentation a 24GB card has two or three GB left for KV cache, while a single full 16K sequence needs 3.75GB — a slightly long prompt and you are out of memory. You can drop to IQ4_XS (18.47GB) or Q3_K_M (16.65GB), but the more honest fix is $0.723/GPU-hr for an RTX 5090 32GB, where the extra 8GB is exactly the room your context needs. If you do want the 4090 at $0.540/GPU-hr, keep it to short conversations, or give it Yi-1.5-9B in bf16 instead.

Why does vLLM only give me a 4096-token context with Yi?

Because max_position_embeddings is literally 4096 in the config.json of Yi-1.5-34B-Chat, Yi-1.5-9B-Chat and Yi-1.5-6B-Chat, and vLLM treats that as the ceiling. This is a long-standing Yi quirk — vLLM has carried an issue about it computing Yi's 200K models as 4K since the early days. The fix is a different repository, not a different flag: use Yi-1.5-34B-Chat-16K or Yi-1.5-9B-Chat-16K (16384) for chat, Yi-1.5-34B-32K or Yi-1.5-9B-32K for a 32K base model, and first-generation Yi-34B-200K if you truly need 200K. Swapping repos and relaunching on NexGPU takes minutes and is billed by the second, so the experiment is cheap.

Yi is Apache 2.0 — is commercial use really unrestricted?

Yes. Yi originally shipped under the Yi Series Models Community License and moved wholesale to Apache 2.0 during 2024, covering both code and weights, for personal, academic and commercial use alike. No email request, no revenue share by user count, no deployment-scale reporting. That is exactly why teams still choose Yi for private deployments: the weights sit on your own disk and the licence is fixed in the file, so whatever the vendor does with its platform cannot reach you. On NexGPU the data inside your instance is likewise yours, and compute billing ends the moment the instance stops.

The open-weight line has stopped updating. Is Yi still worth deploying?

If you want frontier capability, Yi is no longer the first pick — the open weights end at Yi-1.5 (May 2024) and Yi-Coder (September 2024), and the 01-ai repositories on Hugging Face were last updated in November 2024. But Yi-1.5-9B remains a solid Chinese-capable model in the 10B class, Yi-Coder-9B's 128K repository-scale comprehension has almost no sub-10B rival, and the clean licence, plain Llama architecture and zero integration cost make it excellent for the install-it-and-forget-it tier. An RTX 3090 24GB on NexGPU is $0.193/GPU-hr, cheap enough that you can just run your own data through it for an hour and decide on evidence.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.