Skip to main content
NexGPU

Text LLM · Linear Attention / RNN

Self-host RWKV: pure RNN, zero KV cache — 7.2B runs on a single 24GB card

RWKV-7 “Goose” deleted the KV cache outright — take context from 1K to 100K tokens and VRAM does not move by a single byte. The whole G1i line is Apache 2.0, and NexGPU rents the cards by the second from $0.193/GPU-hr.

Sign up freeView pricing

RWKV is not another Transformer variant — it is the branch that went back and got RNNs right. The current production architecture is RWKV-7 “Goose” (paper arXiv:2503.14456, introducing Dynamic State Evolution and a generalized delta rule): linear time, constant space, 100% RNN, no attention anywhere. The project is led by BlinkDL (Bo Peng), joined the Linux Foundation AI & Data in September 2023, is commercialised through Yuanshi Intelligence, and ships every weight under Apache 2.0 — commercial use and closed-source redistribution included. Exactly one line is still being updated: the RWKV7-G1 “GooseOne” reasoning models. The newest drop is G1i, released 2026-08-05 at a training context of ctx16384, in four sizes — 1.5B (L24-D2048), 2.9B (L32-D2560), 7.2B (L32-D4096) and 13.3B (L61-D4096) — all sharing a 65,536-token vocabulary and head size 64. If you are still searching for how to deploy RWKV-4 Raven, RWKV-5 Eagle or RWKV-6 Finch, the official wiki has marked them EOL: swapping to the same-size G1 model is free upside.

RWKV's VRAM arithmetic runs on completely different rules from a Transformer's, which is the real reason to spend an hour of GPU time on it. There is no KV cache; the recurrent state is fixed size. Take 7.2B: L32-D4096 with head size 64 makes the WKV state 32 × 4096 × 64 ≈ 8.4M elements, roughly 34MB in fp32 — and it is still that same 34MB at token 100,000. So your VRAM budget is essentially the weights: G1i .pth files are stored in bf16, so file size is footprint — 3.06GB at 1.5B, 5.9GB at 2.9B, 14.4GB at 7.2B, 26.5GB at 13.3B. Published numbers for 7.2B fp16 on a single RTX 5090: 145+ tok/s decode at bsz1, 11,289 tok/s prefill at bsz1, 9,650+ tok/s decode at bsz320 and 10,250+ tok/s decode at bsz960 — with constant speed and constant VRAM, because going from batch 1 to batch 960 never has to surrender memory to a KV cache. The honest cost sits on the other side: a fixed-size state means recall is lossy, not the exact lookup over full context that attention gives you, so pair it with retrieval for precise long-document extraction.

What actually trips people up is the toolchain, not the VRAM. Start with the big one: vLLM does not support RWKV. Its supported-models list carries Mamba, Mamba2, FalconMamba and Jamba — no RWKV architecture at all — so `vllm serve` is a dead end and copying someone's Qwen recipe will fail. Four paths do work: llama.cpp / Ollama on GGUF (community member @MollySophia landed RWKV v7 support in llama.cpp mainline in March 2025), the official rwkv pip package where a strategy string controls precision directly, transformers plus flash-linear-attention on Triton kernels, and the WebGPU route via Ai00 and web-rwkv. Then there is prompting: RWKV is an RNN and far more format-sensitive than a Transformer. The official form is `User: question\n\nAssistant:` followed by a single space; in Instruction format the Instruction must precede the Input (RWKV's look-back is weaker, so instructions come before material); stray trailing spaces upset the World tokenizer; G1 reasoning is triggered by appending `(think)` to the user prompt; and chat wants temperature 1 with top-p 0.5. The next generation, RWKV-8 “Heron”, is still preview-only — DeepEmbed (May 2025) and ROSA, the Rapid Online Suffix Automaton (October 2025), have experimental weights — so for anything going to production today, G1i is still the answer.

01 —

Current RWKV versions and what they cost in VRAM

G1i shipped 2026-08-05 at ctx16384; .pth files are bf16, so file size is very close to weight VRAM

VersionParametersVRAMContextNotes
RWKV7-G1i 13.3B (rwkv7-g1i-13.3b-20260805-ctx16384)13.3B / L61-D4096bf16 ~26.5GB / Q8_0 ~14.7GB / Q6_K ~11.6GB / Q4_K_M ~8.4GBtrained at ctx16K, extrapolatesThe strongest RWKV you can download. bf16 will not fit a 24GB card — go 48GB, or run Q8_0 on 32GB.
RWKV7-G1i 7.2B (rwkv7-g1i-7.2b-20260805-ctx16384)7.2B / L32-D4096bf16 ~14.4GB / fp16i8 ~9GB / Q8_0 ~7.9GB / Q4_K_M ~4.5GBtrained at ctx16K, extrapolatesThe sweet spot for single-GPU self-hosting, and the size the official RTX 5090 throughput numbers were measured on.
RWKV7-G1i 2.9B (rwkv7-g1i-2.9b-20260805-ctx16384)2.9B / L32-D2560bf16 ~5.9GB / LoRA 8.8GB / state tuning 8.2GBtrained at ctx16KThe size that set a 3B-class multilingual SoTA in the RWKV-7 paper; one 24GB card trains and serves it at once.
RWKV7-G1i 1.5B (rwkv7-g1i-1.5b-20260805-ctx16384)1.5B / L24-D2048bf16 ~3.06GB / LoRA 5.6GB / nf4 tuning 3.9GBtrained at ctx16KThe pure-RNN reasoning model the team benchmarks against Qwen3 1.7B — best value for domain fine-tunes and bulk offline jobs.
RWKV7-G1d 0.4B / 0.1B0.4B / L24-D1024, 0.1B / L12-D768bf16 ~0.9GB / ~0.38GBtrained at ctx8KEdge, embedded, and draft models for speculative decoding. RWKV-7 0.4B trained at ctx4k extrapolates and solves NIAH at ctx16k.
RWKV-8 “Heron” preview (rwkv7a-g1d-0.1b + DeepEmbed)0.1B backbone + DeepEmbed vocabulary vectors2.01GB on disk, but active VRAM stays in the 0.1B classctx8KExperimental. DeepEmbed parks sparse capacity in per-token vectors that cost no VRAM, and ROSA chases genuine infinite context. Not production-ready.

02 —

Which GPU to rent for RWKV

With no KV cache, you size the card to the weights plus a little headroom — longer context never forces an upgrade

  • Evaluating 1.5B / 2.9B, plus LoRA and state tuning

    RTX 3090 24GB$0.193/GPU-hr

    2.9B LoRA in bf16 needs only 8.8GB and state tuning 8.2GB, so one 24GB card both trains and serves — the cheapest way in.

  • Serving 7.2B in bf16 on one card

    RTX 4090 24GB$0.540/GPU-hr

    14.4GB of weights plus a 34MB-class state leaves 24GB with real concurrency headroom — the safest tier for 7.2B in production.

  • High-throughput 7.2B, or 13.3B at Q8_0 / Q6_K

    RTX 5090 32GB$0.723/GPU-hr

    The published 145 tok/s (bsz1) and 10,250+ tok/s (bsz960) figures were measured on a 5090, so you can reproduce them on the same card.

  • 13.3B in full bf16, or sustained large-batch 7.2B training

    RTX A6000 48GB$0.817/GPU-hr

    26.5GB of weights leaves a 32GB card too tight for comfortable batching; 48GB does not. Scale up to H100 SXM 80GB for pretraining.

03 —

Four steps to RWKV running on NexGPU

From cold start to an OpenAI-compatible endpoint — 7.2B usually emits its first token inside half an hour

  1. 01

    Boot a card and get the CUDA kernel environment right

    Pick a PyTorch image from the 2,000+ prebuilt library and start an RTX 4090 24GB. When installing the rwkv pip package, install ninja and set RWKV_CUDA_ON=1 — it compiles the custom CUDA operators, which are faster and use less VRAM. RWKV-7 also requires RWKV_V7_ON=1 explicitly, or the package loads under the old architecture and errors out. Without a C++ compiler or exported CUDA paths it silently falls back to the slow path, so confirm nvcc is visible first.

    pip install rwkv ninja huggingface_hub && export RWKV_V7_ON=1 RWKV_CUDA_ON=1 RWKV_JIT_ON=1 PATH=/usr/local/cuda/bin:$PATH
  2. 02

    Pull the G1i weights

    Official weights live in the BlinkDL/rwkv7-g1 repo on HuggingFace — fetch the exact filename rather than cloning the whole repo. 7.2B is 14.4GB and 13.3B is 26.5GB, which a NexGPU node usually pulls in minutes. If you are going the llama.cpp route, grab a community GGUF instead: 7.2B Q8_0 is only 7.93GB.

    hf download BlinkDL/rwkv7-g1 rwkv7-g1i-7.2b-20260805-ctx16384.pth --local-dir /workspace/rwkv
  3. 03

    Serve it: native pip package or llama.cpp

    The rwkv pip package sets precision with a single strategy string: 'cuda fp16' runs a 7B in about 15GB, 'cuda fp16i8' squeezes it to roughly 9GB, and layered forms like 'cuda fp16i8 *20 -> cuda fp16' let you split precision across layers. Two traps: the model path must omit the .pth extension, and the tokenizer must be the World vocab rwkv_vocab_v20230424. If you want an OpenAI-compatible HTTP endpoint instead, run llama.cpp's llama-server against a GGUF: llama-server -m rwkv7-g1c-7.2b-Q8_0.gguf -ngl 99 --host 0.0.0.0 --port 8080. Official quantisation preference is FP16 > Q8_0 > Q5_K_M > Q4_K_M; anything lower degrades quality sharply.

    python -c "from rwkv.model import RWKV; from rwkv.utils import PIPELINE; m=RWKV(model='/workspace/rwkv/rwkv7-g1i-7.2b-20260805-ctx16384', strategy='cuda fp16'); p=PIPELINE(m,'rwkv_vocab_v20230424'); print(p.generate('User: Explain in one sentence why RWKV needs no KV cache\n\nAssistant:', token_count=200))"
  4. 04

    Fine-tune: state tuning is RWKV's cheap trick

    Use RWKV-PEFT for LoRA, state tuning, MiSS or PiSSA. State tuning trains only the initial state: 1.5B needs 5.3GB in bf16, 4.1GB int8, 3.7GB nf4, and 2.9B needs 8.2GB in bf16 — one 3090 covers it. The resulting state files are tens of MB and hot-swap in Ai00 or RWKV Runner like changing personas. One hard constraint to remember: a state file only works with the exact model version it was trained on, so never mount a G1i-trained state on G1d.

    git clone https://github.com/JL-er/RWKV-PEFT.git && cd RWKV-PEFT && pip install -r requirements.txt

The bill, worked out

Bring up G1i 7.2B in bf16 on an RTX 4090 24GB and benchmark it once: about 12 minutes to boot the image and pull 14.4GB of weights (0.2 hr), 1.5 hr aligning prompt format and debugging state handling, 1.3 hr running bsz1 and bsz64 throughput passes — 3 hours total, 3 × $0.540 = $1.62. Want to reproduce the published 145 tok/s (bsz1) figure on the same silicon the team used? Move to an RTX 5090 32GB: 3 × $0.723 = $2.17. Running 13.3B bf16 (26.5GB of weights) on an RTX A6000 48GB for an 8-hour eval batch: 8 × $0.817 = $6.54. At the cheap end, a 1.5B LoRA run on an RTX 3090 24GB for 6 hours is 6 × $0.193 = $1.16. Storage is separate: weights plus environment is roughly 20GB, so keeping it a month is 20 × $0.414 = $8.28 — destroy the volume when you are done and it stops. Compute is metered per second and priced per hour, billing stops when the instance stops, and there is no minimum, no setup fee and no quota request.

04 —

FAQ

How much VRAM do I actually need to run RWKV locally?

It depends on size and — unusually — not on context length, which is the restful part of RWKV. G1i bf16 weights set the floor: 3.06GB at 1.5B, 5.9GB at 2.9B, 14.4GB at 7.2B, 26.5GB at 13.3B, with a recurrent state of only tens of MB that never grows. Quantised is lower still: 7.2B Q4_K_M is 4.51GB and Q8_0 is 7.93GB. So 7.2B sits very comfortably on a 24GB card, and only 13.3B in bf16 pushes you to 48GB. On NexGPU that is RTX 3090 24GB at $0.193/GPU-hr, RTX 4090 24GB at $0.540/GPU-hr and RTX A6000 48GB at $0.817/GPU-hr, billed by the second — stop when you are done testing.

Can I deploy RWKV with vLLM?

No. vLLM's supported-models list includes Mamba, Mamba2, FalconMamba and Jamba, but no RWKV architecture at all, so `vllm serve` will not work and a copied Qwen deployment script is guaranteed to fail. The four routes that do work: llama.cpp / Ollama on GGUF (RWKV v7 support landed in llama.cpp mainline in March 2025), the official rwkv pip package with its strategy strings, transformers plus flash-linear-attention on Triton kernels, or Ai00 / web-rwkv on WebGPU. NexGPU's image library ships PyTorch, vLLM and Ubuntu CLI ready to go — start from the PyTorch image and installing rwkv or building llama.cpp takes minutes.

RWKV claims infinite context — does long-document work really cost no extra VRAM?

VRAM genuinely does not grow, and that part is not marketing: the state is fixed size. For 7.2B the WKV state is 32 × 4096 × 64 ≈ 8.4M elements, about 34MB in fp32, identical at token 1 and token 100,000. The honest other half: a fixed state means lossy recall, not attention's exact lookup over the whole context, so put retrieval in front of it for tasks like precise clause extraction from long contracts. G1i is also trained at ctx16384 — beyond that you are relying on extrapolation (RWKV-7 0.4B trained at ctx4k solves NIAH at ctx16k), and the further you push, the harder quality is to guarantee. An RTX 4090 24GB at $0.540/GPU-hr on NexGPU lets you map that boundary on your own corpus in an hour or two.

Are RWKV-4 Raven, RWKV-5 Eagle and RWKV-6 Finch still worth deploying?

Not any more. The official wiki marks V4, V5, V6 and even the original V7-World as EOL; the only continuously updated line is RWKV7-G1, whose data versions iterate G1 → G1a → G1a2 → G1b → G1c → G1d and onward to G1i on 2026-08-05, with newer beating older at equal size. If you have an old Raven or Finch deployment, moving to the same-size G1i is usually pure gain — just remember the prompt format and any state files have to move with it. Validating that migration is cheap on NexGPU: an RTX 3090 24GB at $0.193/GPU-hr runs both models against the same eval set for under a quarter an hour.

How many GPUs does RWKV fine-tuning need, and how does state tuning compare to LoRA?

RWKV-7 fine-tuning is abnormally cheap. Published LoRA figures are 1.5B at 5.6GB bf16 / 4.6GB int8 / 3.9GB nf4, and 2.9B at 8.8GB bf16 / 6.7GB int8 / 5.7GB nf4. State tuning is leaner still — 1.5B at 5.3GB bf16, 2.9B at 8.2GB bf16 or 4.7GB nf4 — and the state files it produces are only tens of MB, hot-swappable to change style. In other words, everything up to 2.9B starts on a single card. You only need multi-GPU for continued pretraining at 7.2B and above; the reference configuration is 4×8×H100 at ctx10240 with DeepSpeed zero2 + gradcp, hitting 270k tokens/s at 37% MFU. NexGPU goes up to 14 GPUs per node with a 2,152GB max node VRAM, and H100 SXM 80GB is $3.582/GPU-hr — one 3090 or a multi-node H100 job, same console.

What are the most common mistakes when self-hosting RWKV?

In rough order of how often people hit them: first, prompt format — RWKV is an RNN and much more format-sensitive than a Transformer, the standard being `User: question\n\nAssistant:` followed by one space, with Instruction placed before Input in instruction format, and stray trailing spaces upsetting the World tokenizer (a 65,536-token vocabulary, neither BPE nor sentencepiece). Second, the rwkv pip package wants the model path without the .pth extension, and loading fails outright if RWKV_V7_ON=1 is unset. Third, on the transformers route, flash-linear-attention's Triton kernels are picky about Python and Triton versions — Python 3.10/3.11 have known Triton crashes, so use 3.12 or newer. Fourth, a state file must match the exact model version it was trained on. All of these are fifteen-minute problems to shake out, and NexGPU bills per second with SSH, Jupyter, web terminal, REST API and CLI all open — plus bilingual human support on Telegram with no ticket queue when you get stuck.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.