Skip to main content
NexGPU

Text LLM · Fine-grained Mixture-of-Experts

Self-hosting DBRX: what the 132B MoE actually costs in VRAM

The pay-per-token API is retired and the databricks/dbrx GitHub repo is gone. If you want DBRX today, running it yourself is effectively the only route. bf16 needs 264GB; IQ4_XS drops it to 70.1GB — one A100 80GB and you're live.

Sign up freeView pricing

DBRX is the open-weight model Databricks and the MosaicML team shipped in March 2024, built on a fine-grained MoE design: 132B total parameters with 36B active per token. It uses 16 experts and picks 4, where contemporaries Mixtral-8x7B and Grok-1 used 8 experts picking 2 — Databricks credits the resulting 65x more expert combinations for much of the quality gain. It was pretrained on 12T tokens across 3,072 NVIDIA H100s wired with 3.2Tbps InfiniBand over roughly three months. On benchmarks it posted 73.7% MMLU (5-shot, above GPT-3.5's 70.0% at the time), 70.1% HumanEval (0-shot), 66.9% GSM8k (5-shot) and 66.8% on the 30-plus-task Databricks Gauntlet, while beating Grok-1 at a fraction of its parameter count. Architecturally it is RoPE + GLU + GQA, 40 layers, 48 attention heads, 8 KV heads, the GPT-4 tiktoken vocabulary, and a 32K max context.

Search for DBRX now and you hit three walls. First, on 30 April 2025 Databricks retired DBRX Instruct from Foundation Model APIs pay-per-token, and retired the DBRX, Mistral and Mixtral families from Foundation Model Fine-tuning on the same day — the hosted path is closed. Second, the official databricks/dbrx GitHub repo that carried generate.py, dbrx-full-ft.yaml and dbrx-lora-ft.yaml no longer exists. Third, the original Hugging Face checkpoint was closed, and the transformers documentation now points its example code at transformers-community/dbrx-instruct, a re-upload maintained for compatibility. The good news: the model itself is alive. DbrxForCausalLM is still first-class in transformers, vLLM, llama.cpp and MLX support is intact, and the community GGUF quants are still up. DBRX simply went from a model you could call to a model you have to run.

Which makes GPU selection non-optional here. bf16 weights are 264GB, and the documented practical floor is at least 320GB of memory, distributable across GPUs. Community GGUF quants ladder all the way down: 139.9GB at Q8_0, 108.1GB at Q6_K, 80.0GB at Q4_K_M, 70.1GB at IQ4_XS, 48.0GB at Q2_K, 29.8GB at IQ1_M. Because DBRX's GQA uses only 8 KV heads across 40 layers, a full 32K KV cache costs barely 5GB — almost all your VRAM goes to weights, so "does it fit" is essentially the whole question. NexGPU runs 1,175 verified rentable nodes and 2,498 GPUs across 75 models in 51 countries and regions, up to 14 GPUs per node and 2,152GB max node VRAM, metered per second and priced per hour with no minimum, no setup fee and no quota request. Whether you want to stack 320GB or squeeze IQ4_XS onto a single 80GB card, it's a boot away.

01 —

DBRX variants and what each one costs in VRAM

One size — 132B-A36B — differing only by precision and provenance

VersionParametersVRAMContextNotes
DBRX Instruct132B total / 36B active (16 experts, top-4)bf16 ~264GB; documented floor ~320GB to actually run32K (32,768)The instruction-tuned model — chat, RAG, code Q&A. Knowledge cutoff December 2023.
DBRX Base132B total / 36B activebf16 ~264GB32KUnaligned pretrained base for continued pretraining, custom SFT or paper reproduction. Not a chat model.
transformers-community/dbrx-instruct132B total / 36B activebf16 ~264GB32KThe compatibility re-upload the transformers docs now use in their examples after the original checkpoint was closed. Loads straight into DbrxForCausalLM.
GGUF Q8_0 / Q6_K132B-A36BQ8_0 139.9GB / Q6_K 108.1GB32KNear-lossless tier. The right choice when you want to escape the 264GB bf16 bill without conceding quality.
GGUF Q4_K_M / IQ4_XS132B-A36BQ4_K_M 80.0GB / IQ4_XS 70.1GB32KThe workhorse tier for self-hosting. IQ4_XS measures roughly 5.22-5.25 perplexity on wiki.test and fits every layer onto a single 80GB card.
GGUF Q2_K / IQ2_XS / IQ1_M132B-A36BQ2_K 48.0GB / IQ2_XS 38.5GB / IQ1_M 29.8GB32KExtreme compression. The author of the imatrix quants flags IQ3 and below as very sensitive and unreliable — smoke tests only, never quality evaluation.

02 —

Which GPU to rent for DBRX

VRAM matched to actual weight size — no 'it'll probably squeeze in' recommendations

  • Single-GPU IQ4_XS (70.1GB) for private chat and evals

    A100 PCIE 80GB ×1$0.824/GPU-hr

    70.1GB of weights leaves nearly 10GB free — enough for a full 32K KV cache, and the only genuinely single-card DBRX configuration.

  • Q4_K_M (80.0GB) with more concurrency and long context

    RTX A6000 48GB ×2$0.817/GPU-hr ($1.634/hr total)

    96GB across two cards gives the 80GB checkpoint 16GB of headroom; llama.cpp splits by layer and the bill is an order of magnitude under a single H200.

  • Native bf16 served through vLLM tensor parallelism

    A100 SXM4 80GB ×4$1.088/GPU-hr ($4.352/hr total)

    Four cards hit exactly the 320GB documented floor, and NVLink keeps the MoE all-to-all traffic at TP=4 from becoming the bottleneck.

  • Low time-to-first-token, high-concurrency public serving

    H100 SXM 80GB ×4$3.582/GPU-hr ($14.328/hr total)

    Same 320GB, but the memory bandwidth and FP8 support actually feed the 36B active parameters during decode — for workloads with real QPS pressure.

03 —

Getting DBRX up from scratch

NexGPU ships 2,000+ prebuilt images; vLLM and PyTorch are already there

  1. 01

    Boot, pick an image, pull the weights

    Choose a GPU from the table above, select the vLLM or PyTorch prebuilt image, then get in over SSH or the web terminal. bf16 weights are 264GB, so enable hf_transfer. Note the original databricks/dbrx-instruct is a gated repo requiring an access request; the community re-upload saves you the round trip. Write the weights to storage once and reboots won't re-download them.

    HF_HUB_ENABLE_HF_TRANSFER=1 hf download transformers-community/dbrx-instruct --local-dir /workspace/dbrx
  2. 02

    bf16 multi-GPU: serve an OpenAI-compatible endpoint with vLLM

    TP=4 across four A100 SXM4 80GB. Two flags you must pass explicitly: --trust-remote-code, because DBRX's tokenizer wraps tiktoken, and --max-model-len 32768, because plenty of deployment paths default to a 4K window and you'll otherwise conclude the 32K claim is fiction. If you hit tokenizer errors, check your tiktoken version first — early vLLM builds pinned it to 0.6.0 specifically for DBRX.

    python -m vllm.entrypoints.openai.api_server --model /workspace/dbrx --tensor-parallel-size 4 --max-model-len 32768 --gpu-memory-utilization 0.95 --trust-remote-code
  3. 03

    Single-GPU quantized: IQ4_XS through llama.cpp

    The budget path. IQ4_XS at 70.1GB offloads all 40 layers onto one A100 PCIE 80GB with room left for KV. Max out -ngl, set -c to 32768, and the remaining VRAM covers the cache. DBRX support landed in llama.cpp via PR #6515, so use a recent build — an early-2024 binary won't know the architecture.

    llama-server -m /workspace/dbrx-instruct-IQ4_XS.gguf -ngl 99 -c 32768 --host 0.0.0.0 --port 8080
  4. 04

    Smoke test, then wire it into your stack

    Both paths expose OpenAI-compatible endpoints, so swapping base_url is usually the whole integration. One thing worth writing down: the DBRX model card states it was trained predominantly on English and that non-English proficiency is untested — pilot any non-English workload on a small sample before sending it real traffic.

    curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"/workspace/dbrx","messages":[{"role":"user","content":"Explain fine-grained MoE routing in one paragraph."}]}'

What this actually costs

Here's arithmetic you can audit. Option A: one A100 PCIE 80GB running IQ4_XS for a 2,000-prompt internal eval. Downloading and writing 70.1GB takes about 0.5 hr, llama.cpp load and warm-up 0.2 hr, the eval itself 3.3 hr — 4 hours total, so 4 x $0.824 = $3.296. Option B: native bf16, where 264GB of weights needs four A100 SXM4 80GB to reach 320GB: 4 x $1.088 = $4.352/hr, and an 8-hour team trial is 8 x $4.352 = $34.82. Option C: the same 320GB on four H100 SXM 80GB is 4 x $3.582 = $14.328/hr, or $114.62 for those 8 hours — genuinely faster, and you can decide whether that's worth 3.3x. Don't forget storage: 70.1GB of GGUF at the $0.414/GB-month median is 70.1 x 0.414 = about $29.02/month. Compute billing stops the instant the instance stops; storage keeps running until you destroy the volume. Egress at the $0.0081/GB median makes copying eval results home a rounding error.

04 —

FAQ

How much VRAM does DBRX actually need?

bf16 weights are 264GB, and the documented practical floor is at least 320GB of memory, which can be spread across GPUs. Quantization collapses that: Q8_0 139.9GB, Q6_K 108.1GB, Q4_K_M 80.0GB, IQ4_XS 70.1GB, Q2_K 48.0GB. Because DBRX's GQA uses just 8 KV heads over 40 layers, a full 32K KV cache costs only about 5GB — nearly all VRAM goes to weights, so if it fits, it runs. On NexGPU that maps to one A100 PCIE 80GB at $0.824/GPU-hr for IQ4_XS, or four A100 SXM4 80GB at $1.088/GPU-hr for bf16, both bootable right now.

The official repos are gone — where do I get DBRX weights?

The databricks/dbrx GitHub repo is no longer findable and the original Hugging Face checkpoint was closed. The path of least resistance is transformers-community/dbrx-instruct, the compatibility re-upload the transformers docs themselves now reference; it loads directly into DbrxForCausalLM. For quantized routes, mradermacher/dbrx-instruct-GGUF carries the full Q2_K-through-Q8_0 ladder and dranger003/dbrx-instruct-iMat.GGUF carries imatrix quants from IQ1_S to IQ4_XS. A 264GB download is punishing on home bandwidth; pull it on a NexGPU node with hf_transfer enabled, write it once, and reuse it across reboots.

Can a single RTX 4090 24GB run DBRX?

No. The smallest published quant is IQ1_S at 26.9GB, already past 24GB, and 1-bit output isn't something you should judge a model on. Two 4090s give you 48GB for $1.080/hr, landing exactly on Q2_K's 48.0GB with zero headroom. The interesting part is the arithmetic: a single A100 PCIE 80GB is $0.824/hr — cheaper than the pair, with 32GB more VRAM, enough to run IQ4_XS properly. That's what per-second billing across 75 GPU models on NexGPU is for: picking the right card usually beats stacking cheap ones.

Why does my DBRX only have 4K context when it's supposed to be 32K?

This is the single most common DBRX self-hosting trap. The model really was pretrained with a 32,768-token maximum, but several deployment paths default to 4K and require you to say so: --max-model-len 32768 on vLLM, -c 32768 on llama.cpp. Pass --trust-remote-code too, since DBRX uses the GPT-4 tiktoken vocabulary and goes through the slow tokenizer path; early vLLM pinned tiktoken to 0.6.0 for exactly this reason, so check that version first when tokenizer errors appear. On NexGPU's vLLM prebuilt image it's a two-flag edit and a restart, and the meter stops the moment you stop the instance.

Can I use DBRX commercially? Anything in the license to watch?

It ships under the Databricks Open Model License, which permits commercial and research use with two hard constraints worth reading before you build. Companies above 700 million monthly active users must request a separate commercial license. And you may not use DBRX, its derivatives, or any of its outputs to improve any other large language model — distilling DBRX into your own model is a license violation. The Databricks Open Model Acceptable Use Policy also applies. None of this depends on where you rent compute: NexGPU gives you bare capacity with SSH, Jupyter, web terminal, REST API and CLI access, and your weights and data stay inside your own instance rather than passing through any hosted layer of ours.

In 2026, is DBRX still worth self-hosting?

It depends on why you want it. As a general-purpose assistant, a March 2024 model that its own card describes as predominantly English-trained with untested non-English ability is not where you'd start. But if you're reproducing MoE research, maintaining an existing DBRX pipeline, need a 132B open-weight model with clearly stated license terms as a control, or want to study how fine-grained routing (16 experts, top-4) differs from coarse-grained MoE, DBRX remains a reference point you can't skip — and now that the hosted API is retired, self-hosting is the only door. On NexGPU an A100 80GB is $0.824/GPU-hr metered per second, so a full comparison run costs about what a coffee does, with bilingual support on Telegram and no ticket queue.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.