Text LLM

Before you run Falcon, do the VRAM math

From the 90M Falcon-H1-Tiny to the 68GB bf16 weights of Falcon-H1-34B, from a 665MB BitNet build to a 256K-context hybrid architecture — this page covers only what you actually need in VRAM, and which card to put it on.

Falcon · self-hosted

Falcon is the open model line from TII (Technology Innovation Institute) in Abu Dhabi, but stop judging it by Falcon-180B from 2023. That generation had a 2,048-token sequence length. The current main line, Falcon-H1, puts Transformer attention heads and Mamba-2 state-space heads **in parallel** inside the same mixer block, with the ratio between the two tunable independently. The family spans 0.5B, 1.5B, 1.5B-Deep, 3B, 7B and 34B, each in base and instruct form, with native 256K context and 18 languages that explicitly include Chinese.

The hybrid design pays off directly in the VRAM curve. A pure Transformer's KV cache grows linearly with context length; in Falcon-H1 only a fraction of the layers are attention layers, while the Mamba-2 layers carry a fixed-size recurrent state that does not grow with token count. That is why a 7B model can credibly offer a 256K window, and it is where TII's claim of up to 4x input throughput and 8x generation speedup for Falcon-H1-34B against comparable Transformers comes from. The price of admission is getting the mamba-ssm and causal-conv1d CUDA kernels built, or you silently fall back to a slow path.

Several branches on this line deserve naming individually. Falcon-H1R-7B takes Falcon-H1-7B-Base through SFT on reasoning traces up to 48K tokens plus GRPO reinforcement learning, scoring 88.1% on AIME-24, 83.1% on AIME-25, 68.6% on LiveCodeBench v6 and 61.3% on GPQA-D. Falcon-H1 Arabic ships 3B/7B/34B at 61.87/71.47/75.36 average on the Open Arabic LLM Leaderboard. Falcon-H1-Tiny dropped 15 models between 90M and 600M, including dedicated Coder and Tool-Calling builds. Falcon-E uses BitNet 1.58-bit ternary weights to squeeze 3B down to 999MB. Falcon Perception is a roughly 600M multimodal vision model. The licence is the Falcon LLM License, not Apache 2.0 — only the older Falcon-7B and Falcon-40B are Apache 2.0, so read the attached Acceptable Use Policy yourself before shipping commercially.

01 —

Falcon variants and their real VRAM footprint

Quantised sizes are the actual file sizes in tiiuae's official GGUF repositories, not estimates

Version	Parameters	VRAM	Context	Notes
Falcon-H1-7B-Instruct	7B (~7.6B by weight size)	bf16 15.2GB / Q8_0 8.07GB / Q5_K_M 5.39GB / Q4_K_M 4.6GB / Q2_K 2.89GB	256K	The workhorse tier. At 4.6GB for Q4_K_M a 24GB card still leaves a dozen-plus GB for long context, which makes this the right size to prove things out before committing.
Falcon-H1R-7B	7B	bf16 weights ~14-15GB, FP8 ~7.9GB; add roughly 6GB of KV at 32K context	256K (vLLM default max-model-len 262144)	The reasoning build, emitting long chains of thought. vLLM needs --reasoning-parser deepseek_r1 to split the thinking block from the final answer.
Falcon-H1-34B-Instruct	34B	bf16 ~68GB / Q8_0 35.8GB / Q6_K 27.6GB / Q5_K_M 23.9GB / Q4_K_M 20.3GB / Q3_K_M 16.3GB	256K	The strongest open-weight tier in the family. Q4_K_M at 20.3GB will load on a 24GB card but leaves almost no context headroom; 32GB is where it stops being uncomfortable.
Falcon-H1-1.5B-Deep-Instruct	1.5B (deep and narrow)	bf16 ~3GB / Q8_0 ~1.6GB	256K	TII positions it against current 7B-10B models. Good for bulk offline labelling, routing and classification work that does not deserve a large card.
Falcon-H1 Arabic 3B / 7B / 34B	3B / 7B / 34B	Identical to same-size Falcon-H1 (7B bf16 ~15GB, 34B Q4_K_M 20.3GB)	256K	Current state of the art on Arabic dialect and cultural benchmarks: 61.87 / 71.47 / 75.36 OALL average, with the 34B beating 70B-class systems like Qwen2.5-72B.
Falcon-E-1B / Falcon-E-3B (BitNet 1.58bit)	1.8B / 3B	665MB / 999MB in ternary format	Short-context edge workloads	Small enough for CPU. TII ships both a bf16 build and a pre-quantised BitNet build, the latter designed specifically for cheap fine-tuning.

02 —

Which NexGPU card to rent

Matched honestly to VRAM — we will not point you at a 24GB card for a 34B model at full precision

Falcon-H1-7B at Q4_K_M / Q5_K_M for a single-card smoke test and quality evaluation
RTX 3090 24GB$0.193/GPU-hr
4.6GB of weights plus a 32K llama.cpp context leaves 24GB with plenty to spare, and this is among the cheapest modern CUDA cards per hour on the whole list.
Falcon-H1R-7B in bf16 behind vLLM, serving 64K-context chain-of-thought
RTX 5090 32GB$0.723/GPU-hr
15GB of weights plus 64K worth of KV and Mamba state pushes a 24GB card to its ceiling; 32GB lets you set --max-num-seqs to a concurrency that actually means something.
Falcon-H1-34B at Q4_K_M / Q5_K_M in production on long documents
RTX A6000 48GB$0.817/GPU-hr
Q5_K_M alone is 23.9GB; once you want 128K-class context headroom on top, 48GB is the only single-card option that avoids tensor parallelism entirely.
Falcon-H1-34B in full bf16 for inference or LoRA fine-tuning
A100 SXM4 80GB$1.088/GPU-hr
68GB of bf16 weights has to sit on 80GB of VRAM; if you want the full 256K window or multi-GPU training, nodes go up to 14 GPUs and 2,152GB of node VRAM.

03 —

From empty instance to an OpenAI-compatible endpoint

Every hybrid-architecture trap is in steps one and two — follow these and you will not OOM

01
Boot the instance and install the two kernel packages the hybrid needs
Start from a PyTorch or vLLM prebuilt image so CUDA and drivers are already in place — there are 2,000+ of them. Falcon-H1's Mamba-2 layers depend on mamba-ssm and causal-conv1d, neither of which ships in vLLM's default dependency set. Install them separately with --no-build-isolation, or you will either fail to build or quietly land on a path with no fused kernels.
```
pip install "vllm>=0.11.0" && pip install mamba-ssm causal-conv1d --no-build-isolation
```
02
Serve Falcon-H1R-7B and cap max-model-len by hand
This is the single most common way self-hosted Falcon-H1 falls over: vLLM reads 262144 out of the model config and preallocates KV cache for the full 256K, which OOMs anything from 24GB to 32GB instantly. Set --max-model-len to the length you actually need, then bound concurrency with --max-num-seqs, because every sequence slot holds its own Mamba recurrent state. The reasoning build additionally needs a reasoning parser.
```
vllm serve tiiuae/Falcon-H1R-7B --max-model-len 65536 --max-num-seqs 8 --reasoning-parser deepseek_r1 --gpu-memory-utilization 0.90 --port 8000
```
03
Run the 34B through llama.cpp GGUF on a single card
llama.cpp supports Falcon-H1 natively and the official GGUFs live under tiiuae/Falcon-H1-34B-Instruct-GGUF. Push every layer onto the GPU with --ngl and hold temperature at 0.1 as TII recommends — this model is noticeably sensitive to higher temperatures and quality degrades visibly. If a given quant behaves oddly, cross-check against Q5_K_M or Q6_K; the community has flagged issues on individual tiers.
```
./build/bin/llama-server -m Falcon-H1-34B-Instruct-Q4_K_M.gguf -c 32768 --ngl 999 --temp 0.1 --host 0.0.0.0 --port 8080
```
04
Verify it and point your existing OpenAI client at it
Both paths expose an OpenAI-compatible endpoint, so changing base_url is the whole integration. Reach it over SSH tunnel, Jupyter or the web terminal, or drive instances from CI through the REST API and CLI. Once you have your answer, stop the instance: compute billing stops with it, and only storage keeps accruing until you destroy the volume.
```
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"tiiuae/Falcon-H1R-7B","messages":[{"role":"user","content":"Explain why Mamba-2 state-space layers need no KV cache"}],"temperature":0.1}'
```

What a full evaluation round actually costs

Say you are taking Falcon from zero to production-ready. Smoke test: RTX 3090 24GB at $0.193/GPU-hr x 6 hours = $1.16 to pull Falcon-H1-7B-Instruct Q4_K_M (4.6GB) and check quality on your own long documents through llama.cpp. Load testing: RTX 5090 32GB at $0.723/GPU-hr x 20 hours = $14.46 to move to H1R-7B in bf16 at 64K context and measure concurrency and time-to-first-token under vLLM. Final validation: RTX A6000 48GB at $0.817/GPU-hr x 30 hours = $24.51 to run the 34B Q4_K_M (20.3GB) against real production samples. Compute total: $1.16 + $14.46 + $24.51 = $40.13. Keeping 30GB of weights resident costs 30 x $0.414/GB-month = $12.42. The whole round comes to $52.55, with no minimum rental, no setup fee and no quota request.

04 —

FAQ

How much VRAM does Falcon-H1-7B actually need locally?

It depends which path you take. Through llama.cpp, Q4_K_M is 4.6GB, Q5_K_M 5.39GB and Q8_0 8.07GB — those are the real file sizes in the official GGUF repository. In bf16 you are looking at 15.2GB of weights plus roughly 6GB of KV and state cache at 32K context, so about 22GB in practice. In short, the quantised build is comfortable on a 24GB card while bf16 with long context wants 32GB. On NexGPU that is RTX 3090 24GB at $0.193/GPU-hr and RTX 5090 32GB at $0.723/GPU-hr, metered per second so you stop paying the moment you stop.

Can Falcon-H1-34B run on a single GPU?

Quantised, yes. Q4_K_M is 20.3GB, which loads on a 24GB card but leaves nearly no context headroom. Q5_K_M at 23.9GB and Q6_K at 27.6GB want 32GB or 48GB. Q8_0 at 35.8GB needs 48GB. bf16 at roughly 68GB requires 80GB. Our recommendation is RTX A6000 48GB at $0.817/GPU-hr for Q4/Q5 and A100 SXM4 80GB at $1.088/GPU-hr for bf16 — both available across NexGPU's 1,175 verified rentable nodes.

How does Falcon-H1 relate to Falcon 3 and Falcon 180B, and which should a new project use?

Falcon 180B, Falcon 40B, Falcon 2 11B, Falcon Mamba 7B and Falcon 3 are all still on Hugging Face, but there is little reason to pick them for new work. Falcon-180B had a 2,048-token sequence length while the Falcon-H1 family is natively 256K, and the 7B tier beats earlier large-size Falcons on most benchmarks. For a fresh private deployment, start at Falcon-H1-7B-Instruct, move to Falcon-H1R-7B if you need reasoning, and go to the 34B for maximum quality. On NexGPU those three map to RTX 3090 24GB, RTX 5090 32GB and RTX A6000 48GB respectively.

Why does my Falcon-H1 OOM the moment vLLM starts?

Nine times out of ten, --max-model-len is still at its default. Falcon-H1's config declares 262144 and vLLM preallocates KV cache for the full 256K, which can blow up 24GB and even 80GB cards. Cap it at the length you truly need — 32768 or 65536 — and bound --max-num-seqs, because in a hybrid architecture every concurrent slot holds its own Mamba recurrent state, making concurrency a direct VRAM cost. The other usual cause is mamba-ssm and causal-conv1d not being built correctly. NexGPU's prebuilt vLLM images take most of that compilation pain off the table, with 2,000+ images ready at boot.

How good is Falcon at Chinese, and is it worth a Chinese private deployment?

Falcon-H1 officially supports 18 languages natively and Chinese (zh) is on that list alongside English, Arabic, Japanese and Korean — it is not bolted on by community fine-tunes afterwards. TII's Arabic results, where the 34B tops OALL at 75.36 average and beats Qwen2.5-72B, show the non-English investment is real. Still, test it on your own production samples: that is exactly what per-second billing is for. At $0.193/GPU-hr on an RTX 3090 24GB, a couple of dollars buys you a full blind comparison between Falcon-H1-7B and whatever you run today.

Can I use Falcon commercially, and what is the licence?

The current main line — Falcon-H1, Falcon-H1R, Falcon-H1 Arabic and Falcon-H1-Tiny — ships under the Falcon LLM License, which builds on Apache 2.0 principles but attaches an Acceptable Use Policy. That differs from the common assumption that all Falcon models are Apache 2.0: only the 2023-era Falcon-7B and Falcon-40B are pure Apache 2.0. Read the terms in full before shipping. On the compute side there is no friction to worry about: NexGPU spans 51 countries and regions, 1,175 verified rentable nodes, 2,498 GPUs and 75 GPU models, so you can pick nodes by data residency, with bilingual support over Telegram and no ticket queue. The console lives at console.nexgpu.net.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.