Skip to main content
NexGPU

Text / Multimodal LLM

Self-hosting Kimi starts with one question: how many cards hold 1.56TB?

Moonshot AI has open-weighted everything from K2.5 through K2.7-Code to K3 — and those checkpoints span two orders of magnitude, 16B to 2.8T. Here is the honest VRAM arithmetic for each, and the cards to rent by the hour.

Sign up freeView pricing

Kimi is Moonshot AI's chat product, but engineers searching for "Kimi local deployment" are really after the pile of open weights sitting at huggingface.co/moonshotai. That line is long now: Kimi K2 in July 2025 (1T total / 32B activated, context stretched from 128K to 256K in the September update), K2 Thinking in November, K2.5 in January 2026 adding native multimodality plus instant and thinking modes, K2.6 on 20 April bringing Agent Swarm, K2.7-Code on 12 June aimed squarely at agentic coding, and then K3 with open weights on 27 July — 2.8T total, 104B activated, 1M context.

Each generation rewrites the self-hosting arithmetic. Every release from K2 Thinking onward ships a native INT4 checkpoint, quantised with QAT during post-training rather than bolted on afterwards with GPTQ. That is why a trillion parameters land at roughly 594GB on disk (about 630GB for K2.7-Code) and measure around 640GB of VRAM in practice, on the officially validated 8x H200 141GB, TP=8 configuration. K3 pushes harder: MXFP4 weights with MXFP8 activations, quantisation-aware from the supervised fine-tuning stage onward, and a Hugging Face repo of 96 safetensors shards totalling 1.561TB. That is no longer a "will it fit on one card" question.

So the real question is never "how much VRAM does Kimi need" — it is "which Kimi do you want?" For vision reasoning, Kimi-VL-A3B-Thinking-2506 is 16B under a plain MIT licence and fits on a single 48GB card. To probe KDA linear attention and million-token context, Kimi Linear 48B-A3B needs about 102GB at FP16 — two A100 80GBs. For K2.6's multimodal agent stack, eight H200s is the entry point. For K3 at native precision, budget 1.7TB of aggregate VRAM. NexGPU runs from $0.188/GPU-hour V100s up to 141GB H200s, with up to 14 GPUs and 2,152GB of VRAM in a single node — all four routes are rentable.

01 —

Kimi open-weight releases, side by side

From 16B on one card to 2.8T across a rack, all under the same moonshotai org

VersionParametersVRAMContextNotes
Kimi K32.8T-A104B (896 experts, 16 routed + 2 shared)Native MXFP4 checkpoint 1.561TB; community GGUF bottoms out at UD-IQ1_S 594GB (RAM+VRAM >= 610GB recommended)1,048,576 (1M)Current flagship and the first shipping model built on Kimi Delta Attention (69 KDA layers + 24 Gated MLA). Thinking is always on, image and video input are native. Licensed under a standalone Kimi K3 License, not MIT.
Kimi K2.7-Code1T-A32B (384 experts, 8 routed + 1 shared)Native INT4 around 630GB; needs an 8-GPU box with >=140GB per card (TP=8)256KAgentic coding specialist. Thinking is forced and preserve_thinking stays on, yet it burns roughly 30% fewer thinking tokens than K2.6. Third-party engines do not support video input or instant mode.
Kimi K2.61T-A32B (61 layers, MLA hidden 7168 / 64 heads)Native INT4 around 594GB, roughly 640GB VRAM measured256KThe general-purpose workhorse, with a 400M MoonViT vision encoder and Agent Swarm. Modified MIT, runs on vLLM, SGLang and KTransformers — the best value in the 1T tier for self-hosting.
Kimi K2.51T-A32BNative INT4 around 594GB256KThe first natively multimodal K2, with 15T tokens of vision-language pretraining and switchable instant / thinking modes. Pick it if you need low-latency non-thinking responses — K2.7-Code no longer lets you turn thinking off.
Kimi Linear 48B-A3B-Instruct48B-A3BAbout 102GB at FP16 (1K context) / about 114GB at the full 1M context1MThe public proving ground for KDA: linear attention interleaved with MLA at 3:1, KV cache cut by roughly 75%, decoding up to about 6x faster. The cheapest way to learn how K3's architecture behaves.
Kimi-VL-A3B-Thinking-250616B-A2.8BAbout 32GB at bf16 (budget ~40GB with the vision tower and KV)131,072MIT-licensed vision reasoning model, up to 3.2 million pixels in a single image, 65.2 on VideoMMMU. The only member of the family that genuinely fits on one consumer or workstation card.

02 —

Four Kimi self-hosting scenarios, four honest GPU picks

VRAM matched for real: nobody gets told a 1T model runs on a 24GB card

  • Single card, Kimi-VL-A3B-Thinking-2506 for vision reasoning evals

    RTX A6000 48GB$0.817/GPU-hour

    16B at bf16 is about 32GB; 48GB leaves room for MoonViT's high-resolution tiles and a 128K KV cache without tensor parallelism or extra quantisation.

  • Two cards, Kimi Linear 48B-A3B to test KDA and 1M context

    A100 PCIE 80GB x 2$0.824/GPU-hour

    Published FP16 usage starts at 102GB and reaches about 114GB at the full million-token context; 160GB aggregate covers it with headroom for CUDA graphs and batching.

  • Production Kimi K2.6 / K2.5 / K2.7-Code on the native INT4 checkpoint

    H200 141GB x 8$6.660/GPU-hour

    This is the officially validated 8-GPU TP=8 shape. After roughly 640GB of weights, the remaining 1,128GB of aggregate VRAM goes entirely to the 256K KV pool and concurrency.

  • K3 at native MXFP4 precision on a single node

    H200 141GB x 14$6.660/GPU-hour

    Spread across 14 cards, the 1.561TB checkpoint is about 112GB of weights per GPU; 1,974GB aggregate leaves more KV headroom than the published 16-GPU two-node report (102.75GB weights + 7.95GB KV per card).

03 —

From cold instance to a working Kimi endpoint

K2.6 native INT4 as the main line, with K3's topology and the budget route alongside

  1. 01

    Launch the box and pull the weights — check the repo size first

    Pick an 8x H200 node in the NexGPU console and one of the prebuilt vLLM images (CUDA and PyTorch already in place, from 2,000+ images). K2.6's INT4 weights are about 594GB; K3 is 1.561TB across 96 shards, so size the data volume before you start. Compute is metered per second and stops when the instance stops, but storage keeps billing until you destroy the volume.

    hf download moonshotai/Kimi-K2.6 --local-dir /workspace/Kimi-K2.6
  2. 02

    Serve with vLLM — neither parser is optional

    Kimi's thinking traces and tool calls only come out cleanly through dedicated parsers: drop --reasoning-parser and you never see reasoning_content, drop --tool-call-parser and you get raw text instead of structured calls. Run the multimodal encoder data-parallel — MoonViT's overhead grows as TP goes up. transformers must be >=4.57.1,<5.0.0.

    vllm serve /workspace/Kimi-K2.6 -tp 8 --mm-encoder-tp-mode data --trust-remote-code --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --max-model-len 262144
  3. 03

    Echo reasoning_content back on every multi-turn tool call

    This is the single most common self-hosting failure. In multi-step tool calling, the reasoning_content from the previous assistant message must go back into the context verbatim or the request errors out. Tool call IDs also have to follow the functions.{name}:{idx} format — feed the model a malformed id like search:0 in the history and it will happily generate broken ids from then on. Sample at the recommended temperature 1.0 and top_p 0.95.

    messages.append({"role": "assistant", "reasoning_content": msg.reasoning_content, "tool_calls": msg.tool_calls})
  4. 04

    Budget route: run K3 as a low-bit GGUF under llama.cpp

    If you will not pay for 1.561TB, take the community quants. Unsloth's dynamic ladder runs UD-IQ1_S at 594GB (about 78.9% top-1, PPL 2.58), UD-IQ1_M at 648.9GB (81.2%), UD-Q2_K_XL at 861.3GB (90.4%, PPL 1.74), and UD-Q8_K_XL at 1,560GB for lossless. Rule of thumb: RAM plus VRAM should roughly equal the quant's file size. One warning — Hopper has no native MXFP4, so SGLang falls back to marlin dequantisation; the public report shows 16.8 tok/s single-stream and about 147 tok/s at 54 concurrent requests, with roughly eleven minutes just to load weights. Point Triton's JIT cache at local /tmp, not a shared filesystem, or ranks collide and crash.

    ./llama.cpp/llama-server --model /workspace/K3/UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-*.gguf --mmproj mmproj-BF16.gguf --ctx-size 32768 --temp 1.0 --top-p 0.95 -ngl 99

What running Kimi actually costs

A six-hour private K2.6 evaluation: eight H200s at 8 x $6.660 = $53.28/hour, so $319.68 for the run. Keep the 594GB INT4 checkpoint for three days at 594 x $0.414 / 30 x 3 = about $24.59. Export 20GB of logs and results at 20 x $0.0081 = about $0.16. Total roughly $344.43, and compute billing stops the moment you stop the instance. Want the cheap lane instead? Kimi Linear 48B-A3B on two A100 PCIE 80GBs is 2 x $0.824 = $1.648/hour — a full week of 168 hours costs $276.86, less than those six hours above. And if you insist on K3 at native precision, own the number: 14 H200s is $93.24/hour, and the roughly eleven minutes of weight loading plus 40 seconds of CUDA graph compilation costs about $20.2 on its own under per-second metering. Tune without restarting the server where you can. No minimum, no setup fee, no quota request — those are the only line items.

04 —

FAQ

Can I deploy Kimi locally on a single 24GB GPU?

Not the K2 series or K3 — a trillion-parameter model is 594GB even at native INT4. But the family does have single-card members: Kimi-VL-A3B-Thinking-2506 is 16B, about 32GB at bf16, and a single RTX A6000 48GB at $0.817/GPU-hour carries it with 128K context. If you just want to feel out Kimi's reasoning style on 24GB first, NexGPU's RTX 4090 is $0.540/GPU-hour and the RTX 3090 drops to $0.193, metered per second, stopped whenever you like.

Should I self-host Kimi K3 or Kimi K2.6?

It depends on how many cards you have. K2.6 is 1T-A32B, 256K context, Modified MIT, with a native INT4 checkpoint around 594GB that fits a single eight-card H200 node — the right answer for most teams going private. K3 is 2.8T-A104B with 1M context, a 1.561TB MXFP4 checkpoint, and marlin dequantisation overhead on Hopper. NexGPU supports up to 14 GPUs and 2,152GB of VRAM per node, so both are available; getting K2.6 running on 8x H200 first is simply the cheaper order of operations.

Do Kimi's open-weight licences allow commercial use?

K2, K2.5, K2.6 and K2.7-Code all ship under a Modified MIT licence: standard MIT text plus one clause requiring products above 100 million monthly active users or $20 million in monthly revenue to display the model name prominently. K3 changed that. It ships under a standalone Kimi K3 License with five numbered conditions, including a separate requirement that Model-as-a-Service businesses whose aggregate revenue exceeds $20 million over any consecutive 12 months sign an agreement with Moonshot before commercial use. Have counsel read the actual text rather than reusing the old "Kimi is MIT" shorthand. On NexGPU the instance is yours — weights and data stay on your machine and never touch an inference service of ours.

Why is a 1T model only 594GB? FP16 math says 2TB.

Because Kimi's INT4 is not post-hoc. From K2 Thinking onward, every release applies quantisation-aware training during post-training and Moonshot publishes the native INT4 checkpoint directly: roughly 2x inference speed and half the GPU memory versus FP16, with negligible quality loss. K3 goes further, training with MXFP4 weights and MXFP8 activations from the SFT stage on. So size your cluster from the published checkpoint, not from parameter count times two bytes. Want to verify it yourself? An hour on 8x H200 at NexGPU is $53.28.

Is self-hosting Kimi cheaper than the official API?

On pure token cost, usually not. K2.6's official API runs about $0.95 per million input tokens and $4 per million output; K3 is $3 in and $15 out, dropping to $0.30 on cache hits. Eight H200s cost $53.28/hour, and amortising that takes serious sustained concurrency. Self-hosting pays off elsewhere: data never leaves your VPC, you can keep fine-tuning and distilling on the weights, the version is frozen where you froze it, and latency and routing are yours. NexGPU spans 51 countries and regions with 1,175 verified rentable nodes, so the residency requirement decides the region.

Is Kimi K2 Thinking still worth deploying?

It has been superseded by K2.5 and K2.6, which keep the same 1T-A32B skeleton while adding native multimodality and Agent Swarm, and lift context from K2's original 128K to 256K. Unless your evaluation baseline is pinned to K2 Thinking, start new projects on K2.6 — or K2.7-Code if the work is purely agentic coding, which saves roughly another 30% of thinking tokens. The hardware shape is nearly identical across all of them (8x H200, TP=8), so on NexGPU you swap an image and a weights directory to compare, metered per second, stopped when you are done.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.