Text LLM

Self-host Qwen on a single 24GB card — yes, even the 27B

Qwen3.8-27B at Q4_K_M is 17.1GB of weights, and the hybrid attention stack keeps the KV cache at roughly 2GB for 32K of context. That genuinely fits in 24GB. Below: the VRAM math for every current Qwen variant, which card each one needs, and what it costs per second on NexGPU.

Qwen · self-hosted

Qwen is Alibaba Cloud's model family, published by the Qwen team under the Qwen org on Hugging Face and ModelScope. The open-weight tier is almost entirely Apache 2.0; the -Max and -Plus variants stay closed and API-only. As of August 2026 the line has reached Qwen3.8: Qwen3.8-27B landed on 14 August under Apache 2.0 and is the half you can actually deploy yourself, while Qwen3.8-2.4T-A95B landed on 12 August under a custom Qwen3.8-Max licence that requires a separate commercial negotiation above $50M in annual revenue. Qwen3.7 shipped only as Max and Plus — the open tier was skipped entirely, so if you are hunting for Qwen3.7 open weights, they do not exist.

Starting with Qwen3.5, the architecture switched to a hybrid of Gated DeltaNet linear attention and Gated Attention. In the 27B, only 16 of 64 layers keep a KV cache; the rest are linear layers that keep none. The 9B keeps 8 of 32. The consequence is concrete: 262,144 tokens of native context, extensible to roughly 1M with YaRN, while the KV cache costs about a quarter of what a conventional dense 27B would demand — roughly 2GB at 32K. That is precisely why a 24GB consumer card can hold four-bit 27B weights and a long context at the same time, which was not possible with the equivalently sized models of 2024.

The price is a toolchain that is fussy about versions, and fussy in ways that fail loudly. vLLM needs 0.17 or newer for the GDN kernels. llama.cpp needs a build with llama_memory_hybrid or it refuses the architecture outright. --mamba-cache-mode align is mandatory. And Ampere cards like the A100 have no FP8 tensor cores at all. Working through that on one local card costs you nights. NexGPU spans 51 countries and regions, 1,175 verified rentable nodes, 2,498 GPUs across 75 GPU models, metered per second with no minimum, no setup fee and no quota request, and vLLM and PyTorch are already sitting in the 2,000+ prebuilt images. Retrying on a different card costs cents.

01 —

Qwen variants and what they actually cost in VRAM

Three lines are still maintained in 2026: Qwen3.5, Qwen3.6 and Qwen3.8. Figures are weights only — KV cache is on top.

Version	Parameters	VRAM	Context	Notes
Qwen3.8-27B	27B dense (GDN + Gated Attention hybrid)	bf16 ~54GB / FP8 ~28GB / Q8_0 ~29GB / Q6_K ~22.9GB / Q4_K_M ~17.1GB	262,144 (extensible to ~1M with YaRN)	Released 14 August 2026 under Apache 2.0, the current open flagship dense model. Natively takes image and video input, and exposes reasoning_effort at low / medium / xhigh with xhigh as the default. This is where almost every self-hosting project should start.
Qwen3.8-2.4T-A95B (Qwen3.8-Max)	2.4T total / ~95B active MoE	Not realistically self-hostable — even FP8 lands in the terabyte range	1M	Released 12 August 2026 under a custom Qwen3.8-Max licence rather than Apache 2.0. Unless you have a rack, the API is the realistic path. It is listed here so you do not confuse it with Qwen3.8-27B — they are entirely different propositions.
Qwen3.6-35B-A3B	35B total / 3B active MoE	bf16 ~69GB / FP8 ~35GB / Q4_K_M ~18–22GB	262,144	Released 16 April 2026, Apache 2.0, and the workhorse for agentic coding: 73.4% on SWE-bench Verified and 37.0 on MCPMark. There is an official Qwen3.6-35B-A3B-FP8 using fine-grained FP8 with block size 128, and the quality delta is close to invisible.
Qwen3.5-9B	9B dense	bf16 ~18GB / FP8 ~9GB / Q4_K_M ~5.5GB	262,144	The sweet spot for single-GPU online serving. Only 8 of 32 layers hold a KV cache, and FP8 weights are just 9GB, so on a 24GB card the remaining 15GB all goes to vLLM for KV and continuous batching. Throughput looks very good.
Qwen3.5-4B	4B dense	bf16 ~9GB / Q4 ~3GB (3.4GB as an Ollama pull)	262,144	For edge deployments and batch pipelines. Extraction, classification, rewriting and labelling do not need a bigger model, and this fits comfortably on a Tesla T4 16GB, which keeps unit cost at the floor.
Qwen3.5-122B-A10B	122B total / 10B active MoE (256 experts)	bf16 ~244GB / FP8 ~122GB / Q8_0 ~130GB / Q4_K_M ~77GB	262,144	The practical ceiling when you want near-closed-model quality but must host it yourself. Above it sits Qwen3.5-397B-A17B at roughly 397GB in FP8, which needs a multi-GPU node — and note that Qwen published no official AWQ checkpoint for it, so community quants vary in quality.

02 —

Which NexGPU card to rent

Match the card to the weight format. Do not point a 24GB card at bf16 27B, and never pass --quantization fp8 on an Ampere card.

Get Qwen3.8-27B running at Q4_K_M to validate quality
RTX 4090 24GB$0.540/GPU-hour
17.1GB of weights plus roughly 2GB of KV at 32K context still leaves headroom in 24GB. Ada has native FP8 units, so switching later to Qwen3.5-9B in FP8 (~9GB) needs no new card. On a tighter budget, the RTX 3090 24GB at $0.193/GPU-hour does the same job — a GGUF pipeline never touches FP8 anyway.
Near-lossless Qwen3.8-27B at Q8_0, or a full 128K context window
RTX A6000 48GB$0.817/GPU-hour
Q8_0's 29GB inside 48GB means you never have to shrink the context, and Q6_K at 22.9GB is downright roomy. Note the A6000 is Ampere with no FP8 tensor cores, so run GGUF or AWQ INT4 here and leave --quantization fp8 alone.
Production serving of Qwen3.8-27B or Qwen3.6-35B-A3B in FP8
H100 SXM 80GB$3.582/GPU-hour
27B FP8 is about 28GB and 35B-A3B FP8 about 35GB, so 80GB hands everything left over to KV and continuous batching. Hopper's FP8 is the native target format for these official checkpoints, with no dequantisation step. If bf16 is acceptable, the A100 SXM4 80GB at $1.088/GPU-hour holds the 54GB 27B weights — you simply forgo FP8 speed.
Fit Qwen3.5-122B-A10B in FP8 on one card
H200 141GB$6.660/GPU-hour
The full FP8 weights come to roughly 122GB, and 141GB is the only single-card option that loads them without tensor parallelism, which removes an entire layer of tuning. For Qwen3.5-397B-A17B (~397GB in FP8), take a multi-GPU node — NexGPU goes up to 14 GPUs per node and 2,152GB of node VRAM.

03 —

Four steps to a running Qwen3.8-27B

All of it on one NexGPU instance, over SSH, Jupyter or the web terminal — your pick.

01
Start the instance and get vLLM to 0.17 or newer
The Gated DeltaNet kernels only landed in vLLM main at 0.17. Older builds either fail to recognise the model class outright or limp along behind --trust-remote-code and then blow up on some batch. NexGPU's 2,000+ prebuilt images already carry vLLM and PyTorch, so this is an upgrade rather than an install.
```
uv pip install -U vllm --torch-backend=auto
```
02
Pull the weights
Qwen3.8-27B is roughly 54GB in bf16 and about 28GB for the official FP8 checkpoint. On Asia-Pacific nodes, switch to the ModelScope mirror — the same weights can download several times faster. If you only want a first look, grab GGUF instead: bartowski/Qwen3.8-27B-GGUF ships Q4_K_M at 17.1GB, IQ4_XS at 15.7GB (the largest quant that fits a 16GB card whole) and Q3_K_M at 13.8GB.
```
hf download Qwen/Qwen3.8-27B --local-dir /workspace/Qwen3.8-27B
```
03
Launch vLLM with the GDN-specific flags
--mamba-cache-mode align is mandatory; GDN does not support the all mode. And do not launch at the default 262,144 context on one card — it will OOM every time. Start at 32768, get it stable, then push up. For text-only work, --language-model-only skips the vision encoder and frees another chunk of VRAM. The GGUF equivalent is llama-server -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_M -c 32768 -ngl 99.
```
vllm serve /workspace/Qwen3.8-27B --max-model-len 32768 --mamba-cache-mode align --enable-prefix-caching --reasoning-parser qwen3 --port 8000
```
04
Verify, and switch thinking off while you are there
Qwen3.8 ships with thinking on, so every response opens with a <think>…</think> block. In batch pipelines that is pure wasted output — disable it per request via chat_template_kwargs, or globally at launch with --default-chat-template-kwargs '{"enable_thinking": false}'. To keep some reasoning, dial reasoning_effort to low or medium instead. The field name is enable_thinking: writing thinking instead has triggered accuracy anomalies on some 0-day branches. In thinking mode the official sampling settings are temperature=1.0, top_p=0.95, top_k=20.
```
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"/workspace/Qwen3.8-27B","messages":[{"role":"user","content":"hello"}],"chat_template_kwargs":{"enable_thinking":false}}'
```

What a day of this actually costs

Take the most common path first: Qwen3.8-27B at Q4_K_M on an RTX 4090 24GB, $0.540/GPU-hour. Downloading 17.1GB of weights and setting up vLLM takes about 20 minutes, so 0.33 hours x $0.540 = $0.18. Then run evaluations for 8 straight hours: 8 x $0.540 = $4.32. Total for the day, $4.50. Compute billing stops the moment you stop the instance; only the 17.1GB of weights keep costing, at 17.1 x $0.414/GB-month = $7.08 per month, about 24 cents a day. Want it cheaper? Swap to an RTX 3090 24GB at $0.193/GPU-hour and the same 8 hours is $1.54. | Putting the FP8 build into production: H100 SXM 80GB at $3.582/GPU-hour, running only during business hours at 10 hours a day across 22 working days, is 3.582 x 10 x 22 = $788.04. Full 7x24 residency is 3.582 x 24 x 30 = $2,579.04. | One tier up, Qwen3.5-122B-A10B in FP8 is roughly 122GB of weights and the H200 141GB is the only card that holds it alone, at $6.660/GPU-hour — a full 6-hour evaluation sweep is 6 x $6.660 = $39.96, which is a great deal less than buying the card. Everything is metered per second and priced per hour, with no minimum, no setup fee and no quota request. Median egress is $0.0081/GB.

04 —

FAQ

What is the newest Qwen right now, and is my Qwen3 still fine?

The current line is Qwen3.8: Qwen3.8-27B from 14 August (Apache 2.0, self-hostable) and Qwen3.8-2.4T-A95B from 12 August (custom Qwen3.8-Max licence). Before that came Qwen3.6-35B-A3B and 3.6-27B in April 2026, and the whole Qwen3.5 family in February. Qwen3.7 only ever shipped as closed Max and Plus. Qwen3 from April 2025 still runs, but it tops out at 128K context and gets none of the Gated DeltaNet KV savings — on the same card, Qwen3.8-27B opens a far longer window. Spin up an RTX 4090 on NexGPU and A/B them in twenty minutes rather than guessing.

How much VRAM does Qwen3.8-27B need — is 24GB enough?

It depends on the format. Q4_K_M is 17.1GB, plus roughly 2GB of KV at 32K context, so 24GB is enough and that is the answer for most people. Q6_K at 22.9GB fits too but with no slack. FP8 at 28GB and Q8_0 at 29GB do not fit in 24GB and want 32GB or more, and bf16 at 54GB needs an 80GB card. On NexGPU those tiers map to the RTX 4090 24GB at $0.540, RTX A6000 48GB at $0.817 and A100 SXM4 80GB at $1.088 — billed per second, so getting it wrong and moving to another card costs essentially nothing.

Can an A100 run Qwen in FP8?

No. The A100 is Ampere and has no FP8 tensor cores, so --quantization fp8 either errors out or silently falls back to FP16 — and the silent case is worse, because you assume you saved memory and then OOM. On an A100, use bf16, AWQ INT4 or GGUF. If you genuinely want FP8, use an architecture that supports it natively: Hopper H100 and H200, Ada RTX 4090, Blackwell RTX 5090. NexGPU has all of them on the shelf: H100 SXM 80GB $3.582, H200 141GB $6.660, RTX 4090 24GB $0.540, RTX 5090 32GB $0.723.

Why won't my llama.cpp or Ollama load Qwen3.5 or Qwen3.8?

Because it is a hybrid architecture — some layers are recurrent Gated DeltaNet linear attention, others are standard attention. llama.cpp has to manage both kinds of state through llama_memory_hybrid, and older builds simply refuse to load the model. Upgrade to a build with hybrid memory support. Same story on the vLLM side: 0.17 or newer, and --mamba-cache-mode align is required because GDN does not support the all mode. Note too that prefix caching in align mode is still experimental and generation is not bit-exact between a cached and a fresh run, which matters for regression tests. NexGPU's prebuilt images already have these versions paired up, which saves you a build cycle.

How do I turn off Qwen's thinking mode? The think block wastes output.

Disable it globally at launch with --default-chat-template-kwargs '{"enable_thinking": false}', or per request with chat_template_kwargs: {"enable_thinking": false}. The field must be enable_thinking — people who write thinking instead have hit accuracy anomalies on some 0-day branches, reproduced on both Qwen3.5-27B and 2B. Qwen3.8 adds a second dial: reasoning_effort at low, medium or xhigh, so it is not all-or-nothing. Testing those combinations is exactly what a per-second-billed NexGPU instance is for.

Qwen3.6-35B-A3B only activates 3B parameters — does 8GB of VRAM cover it?

No, and this is the most common MoE misconception. The active count only describes the compute each token traverses; all 35B of weights must sit resident in VRAM, because the router can select any expert at any moment. So size the card by total parameters: bf16 around 69GB, official FP8 around 35GB, Q4_K_M around 18–22GB. MoE saves compute and latency, not memory. On NexGPU that means Q4 on an RTX 4090 24GB at $0.540/GPU-hour, FP8 on an H100 SXM 80GB at $3.582/GPU-hour, bf16 on an A100 SXM4 80GB at $1.088/GPU-hour. The console is at console.nexgpu.net, and support is bilingual over Telegram with no ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.