Skip to main content
NexGPU

Text LLM · Sparse Mixture-of-Experts

Run Mixtral on your own hardware: size VRAM by 46.7B, not 12.9B

Mixtral is a sparse mixture-of-experts model. It activates only 2 of 8 experts per token, so it computes like a 12.9B model — but every weight has to stay resident in VRAM. This page does the honest memory math for 8x7B and 8x22B, then maps it to specific cards.

Sign up freeView pricing

Mistral AI released Mixtral 8x7B in December 2023, and it became the first open MoE model the community actually self-hosted at scale. The architecture is 32 layers, 8 experts per layer, with a router picking the top-2 experts per token: 46.7B total parameters, 12.9B active per token, a 32K context window, Apache 2.0. Mistral's own claim at launch was that it beats Llama 2 70B on most benchmarks with 6x faster inference, and the instruct variant scored 8.30 on MT-Bench — GPT-3.5 territory. Mixtral 8x22B followed with 141B total / 39B active parameters, 56 layers, a 64K context window and native function calling.

One thing to state up front: Mistral's own hosted endpoints, open-mixtral-8x7b and open-mixtral-8x22b, were retired on 30 March 2025, and the official docs now point new integrations at Mistral Small 4. The weights, however, are Apache 2.0 and still sit on Hugging Face — downloadable, commercially usable, redistributable, distillable. Which means self-hosting is now the only way to run Mixtral, and that is precisely why some teams still pick it: an MoE base no vendor can pull out from under you, fully offline, weights you can audit, with a vLLM and llama.cpp ecosystem that has been stable for years.

The first mistake almost everyone makes when self-hosting Mixtral is sizing VRAM off the 12.9B figure. MoE routing is decided per token and per layer — you cannot know which two experts the next token will hit, so all eight experts must stay resident. 46.7B in bf16 is 46.7 × 2 ≈ 93.4GB of weights; 8x22B starts at 281GB. Active parameters govern FLOPs and throughput; they save you exactly zero bytes of memory. Get that arithmetic right and the card choice becomes obvious.

01 —

Which Mixtral variants still exist, and what each one costs in VRAM

Total parameters determine memory. Active parameters determine speed. Never mix the two.

VersionParametersVRAMContextNotes
Mixtral-8x7B-Instruct-v0.146.7B total / 12.9B activebf16 ≈93.4GB / GGUF Q5_K_M 32.23GB / Q4_K_M 26.44GB / AWQ 4-bit 24.65GB32K (32768)The most widely deployed variant, still pulling roughly 400K downloads a month on Hugging Face. Note that its chat template rejects a system role.
Mixtral-8x7B-v0.1 (base)46.7B total / 12.9B activebf16 ≈93.4GB / Q6_K 38.38GB / Q3_K_M 20.36GB / Q2_K 15.64GB32K (32768)The unaligned base for continued pretraining or your own SFT run. Q3_K_M is the only quant that realistically fits a single 24GB card.
Mixtral-8x22B-Instruct-v0.1141B total / 39B activebf16 ≈281GB / GGUF Q5_K_M 99.96GB / Q4_K_M 85.58GB / Q3_K_M 67.78GB64K (65536)56 layers, 48 attention heads, 8 KV heads, v3 tokenizer, native function calling. Clearly stronger than 8x7B on multilingual and math reasoning.
Mixtral-8x22B-v0.1 / v0.3 (base)141B total / 39B activebf16 ≈281GB / IQ3_XS 58.22GB / Q2_K 52.10GB64K (65536)v0.3 is architecturally identical to v0.1; the only change is an extended 32768-token vocabulary. Q2_K fits a single 80GB card but the quality drop is visible.
open-mixtral-8x7b / open-mixtral-8x22b (hosted API)same weights, hosted servicen/a32K / 64KBoth endpoints retired on 30 March 2025; Mistral directs new integrations to Mistral Small 4. Self-hosting is the only remaining path — and the weights are still Apache 2.0.

02 —

Four realistic ways to run Mixtral on NexGPU

VRAM has to hold the full weights plus the KV cache. These are sized against measured file sizes, not optimistic estimates.

  • Single card, 8x7B Q4_K_M, full 32K context

    RTX 5090 32GB$0.723/GPU-hour

    Q4_K_M weights are 26.44GB and 8x7B's KV cache runs 128 KiB/token, so a full 32K context adds exactly 4GB — 30.4GB total fits 32GB with room to spare, while a 24GB card forces you down to Q3_K_M and a truncated context.

  • Single-card vLLM OpenAI-compatible endpoint, AWQ 4-bit, real concurrency

    RTX A6000 48GB$0.817/GPU-hour

    AWQ weights take only 24.65GB, leaving ~20GB for the KV cache pool — about 160K tokens at 128 KiB/token, enough for five concurrent 32K sessions without touching tensor parallelism.

  • 8x7B at full bf16 precision in production, TP=2

    A100 PCIE 80GB × 2$0.824/GPU-hour ($1.648/hr for the pair)

    93.4GB of weights split across tensor-parallel-size 2 is 46.7GB per card, leaving roughly 30GB each for KV cache and activations — the cheapest configuration with zero quantisation loss.

  • 8x22B Q4_K_M inference at 64K context

    A100 SXM4 80GB × 2$1.088/GPU-hour ($2.176/hr for the pair)

    85.58GB of weights plus 14GB of KV cache at full 64K context sits comfortably in 160GB; if you insist on a single unsharded card, only the H200 141GB ($6.660/GPU-hour) will hold it.

03 —

From boot to first token: four steps to a working Mixtral

Copy-paste ready. The mistralai repos are gated, so authenticate before you pull.

  1. 01

    Launch the instance and pull the weights

    Pick an image with PyTorch or vLLM preinstalled, SSH in, and log into Hugging Face first. Every repo under the mistralai org is gated: you must accept the terms on the model page and authenticate with a token, or you get 'Cannot access gated repo' immediately. The 8x7B bf16 weights are about 93.4GB and typically land in tens of minutes on NexGPU node bandwidth.

    huggingface-cli login && huggingface-cli download mistralai/Mixtral-8x7B-Instruct-v0.1 --local-dir /workspace/mixtral-8x7b
  2. 02

    Serve it with vLLM as an OpenAI-compatible endpoint

    Run tensor parallelism across two 80GB A100s. vLLM's Mixtral path shards each expert across all ranks and runs a fused MoE kernel in the forward pass; if you later scale to multi-replica serving, add --enable-expert-parallel to switch the expert layers to expert parallelism instead. For a quantised deployment, point at the AWQ repo and add --quantization awq.

    vllm serve /workspace/mixtral-8x7b --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 32768 --gpu-memory-utilization 0.90
  3. 03

    Or go single-card with llama.cpp / Ollama

    With one 32GB or 48GB card, use GGUF. 8x7B has 32 layers, so -ngl 33 pushes every layer plus the output head onto the GPU — leave any layer on the CPU and MoE expert swapping will collapse your throughput. For the lazy path, ollama run mixtral:8x7b-instruct-v0.1-q4_K_M does the same thing. One note: the RTX 5090 is Blackwell, so the PyTorch side needs a CUDA 12.8+ build; llama.cpp has no such constraint.

    ./llama-server -m mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf -ngl 33 -c 32768 --host 0.0.0.0 --port 8080
  4. 04

    Work around the missing system role

    Mixtral v0.1's official chat template knows only user and assistant — the [INST] ... [/INST] format has no slot for a standalone system message. Pass a system role and you get TemplateError: Conversation roles must alternate user/assistant. The safest fix is prepending your system prompt to the first user message. Alternatively use mistral-common, which folds the system content into the last user message by default, or hand vLLM a --chat-template that supports system messages.

    messages = [{'role': 'user', 'content': system_prompt + '\n\n' + user_query}]

A real invoice: what fine-tuning and serving Mixtral actually costs

Start with QLoRA fine-tuning of Mixtral 8x7B. At 4-bit the training state needs roughly 32GB of VRAM, so one A100 PCIE 80GB at $0.824/GPU-hour is comfortable. Pulling the 93.4GB of weights at ~40 minutes: 0.67 × $0.824 ≈ $0.55. Three epochs over 22 hours: 22 × $0.824 = $18.13. Weights plus checkpoints occupy 120GB of storage held for two days: 120 × $0.414 ÷ 30 × 2 ≈ $3.31. Exporting a 1.2GB LoRA adapter: 1.2 × $0.0081 ≈ $0.01. Total ≈ $22.00. Stop the instance when training ends and compute billing stops on the spot; storage keeps accruing until you destroy the volume. Now inference: 8x7B Q4_K_M is 26.44GB, so a four-hour evaluation on an RTX 5090 32GB is 4 × $0.723 = $2.89. Move to 8x22B Q4_K_M (85.58GB) and two A100 SXM4 80GB cards cost $1.088 × 2 = $2.176/hr, so the same four hours is $8.70. One pricing detail worth catching: the RTX A6000 48GB is $0.817/GPU-hour and the A100 PCIE 80GB is $0.824 — seven-tenths of a cent an hour apart, for 32GB more VRAM and far more bandwidth. For Mixtral fine-tuning there is essentially no reason to pick the smaller card. Everything is metered per second and priced per hour, with no minimum, no setup fee and no quota request.

04 —

FAQ

How much VRAM does Mixtral 8x7B actually need?

It depends which tier you run. Full bf16 is 46.7 × 2 ≈ 93.4GB of weights plus about 4GB of KV cache at full 32K context, which means two 80GB cards at TP=2. The quantised tiers are measured file sizes: GGUF Q5_K_M 32.23GB, Q4_K_M 26.44GB, Q3_K_M 20.36GB, Q2_K 15.64GB, and AWQ 4-bit at 24.65GB. For a single card running the full 32K context, 32GB is the lowest comfortable line. NexGPU lists the RTX 5090 32GB at $0.723/GPU-hour and the A100 PCIE 80GB at $0.824/GPU-hour, billed per second — spin one up for an hour, measure your own tier, then commit.

If only 12.9B parameters are active, why does it need 93GB of VRAM?

Because the routing is dynamic. Each of Mixtral's 32 layers holds 8 experts and the router picks the top-2 per token — and which two changes per token and per layer independently. You cannot predict the next token's path, so all eight experts must stay resident. The 12.9B figure governs FLOPs and throughput (which is why Mixtral serves at roughly 13B-model speed) and has nothing to do with memory footprint. This is the single most common MoE misjudgement, and the reason people rent the wrong card. On NexGPU you size by total parameters and pay by the second, so if you do get it wrong you stop the instance and switch — nothing is locked into a monthly plan.

Is Mixtral deprecated? Is self-hosting it still worth it?

What was deprecated is Mistral's hosted API: open-mixtral-8x7b and open-mixtral-8x22b were both retired on 30 March 2025, and the docs now route new integrations to Mistral Small 4. The weights are Apache 2.0 and remain on Hugging Face — downloadable, commercially usable, redistributable. So self-hosting is the only way to run Mixtral today, and that is exactly the appeal: an MoE base no provider can retire out from under you, fully offline, with auditable weights. Pick a NexGPU instance with a vLLM image and you have your own OpenAI-compatible endpoint in about fifteen minutes.

Can Mixtral 8x22B run on a single GPU?

Not at bf16 — 281GB is out of reach for any single card. Quantised, Q4_K_M is 85.58GB, and adding 14GB of KV cache for the full 64K context brings you to roughly 100GB, which only an H200 141GB will hold on one card ($6.660/GPU-hour on NexGPU). Q2_K at 52.10GB does fit a single 80GB A100, but 2-bit quantisation hurts MoE quality noticeably and the community generally advises against it. The better economics are two A100 SXM4 80GB cards at TP=2: $1.088 × 2 = $2.176/hr. NexGPU nodes take up to 14 GPUs with a maximum node VRAM of 2,152GB, so even full bf16 8x22B has somewhere to land.

How good is Mixtral at Chinese and other non-listed languages?

Honestly: Mistral officially lists English, French, Italian, German and Spanish — Chinese is not among them. In practice Mixtral reads Chinese fine, but fluency, Chinese instruction-following and long-form consistency are visibly weaker than Chinese-native bases from the same period. For a Chinese-language product you generally need another round of Chinese SFT on top of Mixtral, or a different base entirely. Don't take anyone's word for it: at $0.723 an hour on an RTX 5090 32GB, billed per second, you can run Q4_K_M against your own eval set on NexGPU and have the answer within the hour.

I get TemplateError: Conversation roles must alternate user/assistant. How do I fix it?

This is Mixtral's most frequent deployment error. The official v0.1 chat template recognises only user and assistant; the [INST] ... [/INST] structure has no slot for a standalone system message, so passing a system role raises this exception. Three fixes: prepend the system prompt to the first user message (one line, simplest); switch to mistral-common, which merges system content into the last user message by default; or pass vLLM a --chat-template pointing at a custom template that supports system messages. Environment friction like this is best cleared once inside a prebuilt image — NexGPU ships 2,000+ images including PyTorch, vLLM and Jupyter, with SSH, web terminal, REST API and CLI access, and bilingual support on Telegram with no ticket queue.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.