Text LLM

Self-hosting GLM: one 4090 at the bottom, eight H200s at the top

From the 30B-A3B GLM-4.7-Flash to the 744B GLM-5.2, one model family spans a fortyfold difference in VRAM. Get the arithmetic right before you rent anything.

GLM · self-hosted

GLM is Zhipu's open-weight family, shipped internationally under the Z.ai brand. Weights live in the zai-org organisation on Hugging Face, the main repo is github.com/zai-org/GLM-5, and the whole line is MIT-licensed — commercial use allowed, weights modifiable, no regional restrictions, no supplementary agreement to sign. That is rare at the frontier, and it is exactly why GLM keeps showing up in private deployments: no quota request, no approval queue, just `hf download` and it runs on your own cards.

The current line is the GLM-5 family. GLM-5.2 is a sparse MoE with 744B total and 40B active parameters, 78 layers at a hidden size of 6,144, using MLA stacked with DeepSeek Sparse Attention plus a layer called IndexShare — one indexer is shared across every four layers, with the following three reusing the top-k indices the first one selected, cutting per-token FLOPs at 1M context to roughly 1/2.9. Native context window is 1,048,576 tokens. GLM-5.3 is a post-training upgrade on that same base with no new pre-training, aimed at coding and long-horizon agents; it launched API-first, with open weights following on the zai-org repos roughly two weeks later. So if you need to stand up a service right now, the newest downloadable tier is still GLM-5.2 and its FP8 checkpoint. GLM-4.6 (355B-A32B, 200K context) is still in the repos but has been superseded outright by GLM-4.7.

The other branch is the models that genuinely fit on one card. GLM-4.7-Flash is a lightweight 30B-A3B MoE (Glm4MoeLiteForCausalLM) with a 202,752-token context; UD-Q4_K_XL is just 17.5GB, so a 24GB card has room to spare. Above it sits GLM-4.5-Air (106B-A12B, 131,072 context) at roughly 66.7GB for Q4_K_M, which means two 48GB cards minimum. That gap is the painful part of self-hosting: Flash runs on consumer silicon, the flagship wants eight H200s, and most people just want to validate one configuration before committing. On NexGPU both ends of that ladder are available on demand, metered per second and priced per hour, with no minimum rental period, no setup fee, and compute billing that stops the moment the instance stops.

01 —

GLM versions, parameters, and real VRAM footprints

One family spanning 30B to 744B — pick the wrong tier and you have wasted a night of GPU time

Version	Parameters	VRAM	Context	Notes
GLM-5.3	744B total / 40B active (same base as GLM-5.2)	Weights not yet published; by the same base, FP8 would be ~744GB	1M	A pure post-training upgrade — Z.ai's internal evals put coding roughly 50% above GLM-5.2, with 84.5% on CyberGym. The zai-org/GLM-5.3 repo is still gated, so it is API-only for now; stand your stack up on GLM-5.2 and the serving flags carry over almost unchanged when the weights land.
GLM-5.2 / GLM-5.2-FP8	744B total / 40B active	FP8 ~744GB \| BF16 ~1.51TB \| GGUF UD-Q4_K_XL 467GB \| UD-IQ4_XS 365GB \| UD-Q2_K_XL 254GB \| UD-IQ1_S 217GB	1,048,576	The strongest open weights you can actually download today. MLA + DSA + IndexShare keep 1M context affordable, and an MTP layer ships for speculative decoding. Requires SGLang v0.5.13.post1+ or vLLM v0.23.0+. NVIDIA also publishes an NVFP4 checkpoint that runs TP=8 on B200/B300 with benchmark scores essentially level with FP8.
GLM-4.7 / GLM-4.7-FP8	358B total / 32B active	BF16 ~716GB \| FP8 ~358GB	202,752	92 layers (3 dense + 89 MoE), 160 routed experts with top-8 activation plus 1 shared expert. Introduced Preserved Thinking, Turn-level Thinking, and Interleaved Thinking; scores 42.8% on HLE, a 12.4-point gain over GLM-4.6. The reference serving config is 4x-8x H200 running FP8.
GLM-4.7-Flash	30B-A3B	UD-Q4_K_XL 17.5GB \| Q4_K_M 18.3GB \| Q6_K 24.7GB \| Q8_0 31.8GB \| BF16 59.9GB \| UD-Q2_K_XL 11.9GB	202,752	The only genuine single-card option in the family, positioned by Z.ai as the strongest model in the 30B class for local coding and agents. Roughly 3B active parameters per token means low memory pressure and fast generation — tens of tokens per second on a 24GB card at Q4. Sampling: temperature 1.0 / top_p 0.95 in general, temperature 0.7 / top_p 1.0 for tool calling, and repeat penalty off.
GLM-4.5-Air	106B-A12B	Q4_K_M ~66.7GB (add ~20GB of KV cache at 128K context)	131,072	The previous generation's mid-weight option: 46 layers, 128 experts per MoE layer with top-8 routing. Good when Flash is not enough but an eight-card cluster is overkill. KV cache grows linearly with context — about 1.3GB at 8K, 5.0GB at 32K, 20GB at 128K — and you must budget for it.

02 —

Four realistic ways to run GLM on NexGPU

List rates are per GPU per hour, metered per second, and compute billing stops when the instance stops

Single card running GLM-4.7-Flash at Q4 as a local coding assistant or a first evaluation
RTX 4090 24GB$0.540/GPU-hr
UD-Q4_K_XL is 17.5GB, leaving over 6GB for KV cache — comfortable at 32K-64K context, and the cheapest sane entry point into the GLM line.
GLM-4.7-Flash at Q8_0 or BF16 for a precision comparison, or two cards for GLM-4.5-Air at Q4
RTX A6000 48GB$0.817/GPU-hr
48GB swallows the 31.8GB Q8_0 checkpoint with long-context headroom left over; two cards give 96GB, which covers GLM-4.5-Air's 66.7GB of weights plus 128K of KV cache.
Full private GLM-4.7 FP8 deployment serving 200K-context inference
H100 SXM 80GB x 8$3.582/GPU-hr ($28.656/hr for eight)
FP8 weights are around 358GB, so 640GB across eight cards leaves over 200GB for KV cache and concurrency — and Hopper's native FP8 tensor cores consume the official FP8 checkpoint without re-quantising.
Single-node GLM-5.2 FP8 with the 1M context window actually in use
H200 141GB x 8$6.660/GPU-hr ($53.28/hr for eight)
744GB of FP8 weights will not fit in eight H100s' 640GB. Eight H200s give 1,128GB — the smallest single-node config that holds the weights and still leaves KV room — and TP=8 starts in one command with no cross-node fabric to configure.

03 —

Four steps to a running GLM endpoint

From boot to correctly parsed tool calls — nearly every trap lives in the serving flags

01
Boot an instance with vLLM already on it
Pick a GPU at console.nexgpu.net and launch from a prebuilt vLLM or PyTorch image — there are 2,000+ of them, which saves you compiling CUDA and chasing dependencies. One RTX 4090 24GB for GLM-4.7-Flash; the eight-card H200 node shape for GLM-5.2 FP8. Get in over SSH, Jupyter, or the web terminal. Size the disk at about 1.3x the weight size, since GGUF quants and original safetensors will sit side by side.
```
ssh root@<your-instance> -p <port>
```
02
Pull the weights
The entire GLM line is MIT-licensed, so there is no access request on Hugging Face — just download. Mind which repo you want: BF16 is zai-org/GLM-4.7, FP8 is zai-org/GLM-4.7-FP8, and the GGUF quants live in unsloth's mirror repos. GLM-5.2's FP8 checkpoint is close to 744GB, so confirm disk and bandwidth before you start rather than halfway through.
```
hf download zai-org/GLM-4.7-Flash --local-dir ./GLM-4.7-Flash
```
03
Serve it, and get the three parser flags right
This is where GLM self-hosting most often goes wrong. GLM-4.7 and the whole GLM-5 series emit the newer `<tool_call>…<arg_key>…<arg_value>` format, so `--tool-call-parser glm47` is mandatory. The older glm45 parser will not error — it will silently leave the tool call as raw text inside message.content, which looks exactly like a model that cannot call tools. Reasoning traces still use `--reasoning-parser glm45`. If you enable MTP speculative decoding, one speculative token gives 90%+ acceptance and the best throughput on GLM-4.7. Note that at launch both vLLM and SGLang only supported GLM-4.7 on their main branches, so upgrade to nightly if the architecture name is unrecognised.
```
vllm serve zai-org/GLM-4.7-Flash --tensor-parallel-size 1 --max-model-len 65536 --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice
```
04
For the GGUF route, use llama.cpp and check the sampler
On a 24GB card, llama.cpp is the path of least resistance for Flash. Two defaults must change: set `--min-p 0.01` (the 0.05 default prunes candidates noticeably), and set repeat penalty to 1.0 or disable it outright, or output quality collapses. One historical trap: early llama.cpp builds set GLM-4.7-Flash's `scoring_func` to softmax when it should be sigmoid, which caused endless looping — after the fix you need to re-download the GGUFs. Ollama's chat-template handling for this GGUF is not reliable and is not recommended.
```
llama-server -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL --ctx-size 16384 -ngl 99 --temp 1.0 --top-p 0.95 --min-p 0.01 --repeat-penalty 1.0
```

What two tiers of GLM actually cost

Start light. GLM-4.7-Flash at UD-Q4_K_XL (17.5GB) on one RTX 4090 24GB is $0.540/GPU-hr. Run it for 8 straight hours of prompt tuning and tool-calling regression: 8 x $0.540 = $4.32. Keep the weights on disk at 17.5GB x $0.414/GB-month = $7.25/month, about $0.24/day, so three days is $0.72. Total for the whole three-day exercise: roughly $5.04 — less than many API bills for a single day, with the model entirely on your own machine. Now the heavy tier. GLM-5.2 in FP8 needs eight H200 141GB: 8 x $6.660 = $53.28/hr. Budget 0.5 hours to download and warm up, then 5.5 hours of load testing and long-context evaluation, for 6 hours total: 6 x $53.28 = $319.68. Park the 744GB checkpoint in storage for two days: 744 x $0.414 = $307.94/month, about $10.26/day, so $20.52. Export 20GB of eval logs and sampled outputs: 20 x $0.0081 = $0.16. Total around $340.36 to find out exactly how a 744B model behaves at 1M context on your workload. One thing to watch: compute billing stops the instant the instance stops, but storage keeps accruing until you destroy the volume. If you are done with the evaluation for now, delete the weight volume and that $10.26/day goes to zero.

04 —

Frequently asked questions

What is the minimum VRAM for running GLM locally?

It depends entirely on which GLM. The floor is GLM-4.7-Flash at UD-Q2_K_XL, just 11.9GB, which fits a 16GB card; the practical starting point is UD-Q4_K_XL at 17.5GB on a 24GB card. Above that, GLM-4.5-Air at Q4_K_M wants about 66.7GB, GLM-4.7 in FP8 about 358GB, and GLM-5.2 in FP8 about 744GB. Do not try to force a 358B model onto a 24GB card — once it offloads to system RAM the speed becomes unusable. NexGPU runs 75 GPU models and 2,498 cards from Tesla T4 16GB ($0.298/GPU-hr) up to H200 141GB ($6.660/GPU-hr), so every rung of that ladder is one click away.

Can a single RTX 4090 run GLM?

Yes — GLM-4.7-Flash. It is a 30B-A3B MoE activating roughly 3B parameters per token, and UD-Q4_K_XL weighs 17.5GB, so a 4090's 24GB leaves over 6GB for KV cache and 32K context is comfortable. The flagships are out of reach: 358B and 744B of weights are simply not a single-card proposition. NexGPU lists RTX 4090 24GB at $0.540/GPU-hr, and if you want cheaper, RTX 3090 24GB is $0.193/GPU-hr with the same VRAM and one tier less bandwidth and compute.

When can I download GLM-5.3's open weights, and which version should I self-host now?

GLM-5.3 is a post-training upgrade on the same base as GLM-5.2 with no new pre-training. It launched API-first, with open weights following on the zai-org Hugging Face org roughly two weeks later; before that the repo is gated and a direct pull returns 401. So the newest downloadable tier right now is GLM-5.2 and GLM-5.2-FP8. The good news is that sharing a base means serving flags and parser configuration carry straight over — swapping the model path is most of the migration. On NexGPU you can spin up eight H200s metered per second, get your whole serving pipeline and load tests working against GLM-5.2, and swap models the day the weights land.

Why are my GLM tool calls showing up as plain text instead of being parsed?

Almost certainly the wrong parser. GLM-4.7 and the entire GLM-5 series emit the newer `<tool_call>…<arg_key>…<arg_value>` structure, which requires `--tool-call-parser glm47`. The older glm45 parser throws no error but leaves the call verbatim in message.content. Two related known issues: tool calls could break while MTP speculative decoding was active until a vLLM fix landed, so upgrade; and if you see "tool call result is empty", add `--chat-template-content-format=string`. Version-sensitive debugging like this is cheap when you are billed by the second — on NexGPU you can shake out the serving flags on a single 4090 for a few dollars before moving to the big node.

Why does self-hosted GLM-5.2 burn so many tokens?

Because its chat template resolves reasoning_effort to max by default, not the high that most people assume — high is actually the opt-in setting. The default tier spends a great deal of budget on reasoning traces, which is especially visible on long-context tasks. Pass reasoning_effort explicitly at the level you actually want and output lengths drop back to normal immediately. This is one of the underrated advantages of self-hosting: wasted tokens show up directly as GPU hours, and because NexGPU meters per second, you can see at a glance which configuration is burning card time.

GLM-4.7-Flash keeps looping in llama.cpp — how do I fix it?

First check whether your GGUF is stale. Early llama.cpp builds set GLM-4.7-Flash's `scoring_func` to softmax when the correct value is sigmoid, which causes endless looping; after the fix you have to re-download the weight files. Then check sampling: repeat penalty must be 1.0 or disabled, `--min-p` should be set explicitly to 0.01 to override the 0.05 default, and use temperature 1.0 / top_p 0.95 generally, switching to temperature 0.7 / top_p 1.0 for tool calling. Ollama is also not recommended for this particular GGUF because of chat-template compatibility. If you want to reproduce it in a clean environment, NexGPU has 2,000+ prebuilt images and bilingual support over Telegram with no ticket queue, so you can just ask about a serving flag that has you stuck.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.