Skip to main content
NexGPU

Open-Weight LLM

Deploying Llama locally starts with doing the VRAM math

From Llama 3.2 1B at 808MB to Maverick's 400GB of FP8 weights, one name spans a 500x hardware gap. Here is what each variant actually costs in VRAM, the commands to serve it, and which card to rent.

Sign up freeView pricing

Llama is Meta's open-weight family, running from February 2023 through to Llama 4 Scout and Maverick on 5 April 2025. One thing to state plainly up front: Meta has not shipped a new Llama since. In April 2026 Meta Superintelligence Labs pivoted to the closed-weight Muse Spark, and in August 2026 released the 30B Muse Glimmer under Apache 2.0. The naming line stopped at Llama 4 — but not a single weight was withdrawn. Llama-4-Scout-17B-16E-Instruct and Llama-3.3-70B-Instruct are still on Hugging Face under the meta-llama org, with a tooling ecosystem deeper than anything newer.

What really drives your card choice is Llama 4's MoE architecture. Scout is 17B active parameters across 16 experts for 109B total; Maverick is 17B active across 128 experts for 400B total. Inference only computes 17B, but every expert has to sit resident in VRAM. That makes the official claims of "Scout runs on a single H100" and "10M token context" mutually exclusive: vLLM's own blog gives 8xH100 with tensor parallelism to reach 1M context, and 8xH200 to reach 3.6M. It is also why the most-deployed open-weight models in production today are still Llama 3.1 70B and Llama 3.3 70B, not Llama 4.

On licensing, the whole family ships under the Llama Community License, which does not meet the OSI open-source definition — open-weight is the accurate term. Products above 700 million monthly active users must request a separate grant that Meta may refuse; you must display "Built with Llama" prominently; any model you fine-tune from it must have a name beginning with "Llama"; and the multimodal variants explicitly withhold the grant from companies with a principal place of business in the EU. None of that stops you running it on your own machines — it only shapes how you ship. And self-hosting has exactly one hard gate, which is whether the VRAM adds up. On NexGPU that ranges from an RTX 3090 24GB at $0.193/GPU-hr to an H200 141GB at $6.660/GPU-hr, metered per second, no minimum, no quota request.

01 —

Llama VRAM reference table

Weight sizes taken from official Ollama tags and Unsloth quantization repos — measured, not estimated

VersionParametersVRAMContextNotes
Llama 3.2 1B / 3B Instruct1B / 3BQ4_K_M 808MB / 2.0GB; fp16 2.5GB / 6.4GB128KThe edge tier. 3B is the cheapest usable chat backbone going — a Tesla T4 16GB runs several streams at once. Usually deployed for intent classification, request routing and structured extraction, jobs that do not need a big brain.
Llama 3.1 8B Instruct8BQ4_K_M 4.9GB; q8_0 8.5GB; fp16 16GB128KThe tier with the deepest fine-tuning ecosystem — LoRA scripts, quant formats and inference backends all target it by default. 16GB of fp16 weights plus a 32K KV cache fits a single 24GB card, making it the natural starting point for self-hosted RAG.
Llama 3.2 11B / 90B Vision Instruct11B / 90B11B bf16 ~22GB; 90B needs 64GB+128KMultimodal variants built by bolting a vision adapter onto the Llama 3.1 text backbone. Good for OCR, chart QA, and invoice or contract extraction. The 11B is tight on 24GB and comfortable on 32GB; the 90B wants 80GB and up.
Llama 3.3 70B Instruct70BQ4_K_M 43GB; q8_0 75GB; fp16 141GB128KReleased December 2024, it reaches roughly Llama 3.1 405B quality at 70B of volume. It is the single most widely deployed open-weight model out there, hosted by 20+ API providers. If you are self-hosting Llama on a new project, this is the default.
Llama 4 Scout 17B-16E Instruct17B active / 109B totalINT4 ~55GB; IQ1_S 33.8GB; Q2_K_XL 42.2GB; Q4_K_XL 65.6GB; bf16 ~210GB10M nominal, KV-cache bound in practiceThe first natively multimodal Llama. Meta claims INT4 fits a single H100 — it does load, but the KV cache runs dry fast. The community also reports silent hangs past ~3M tokens, and this model does not support torch.compile.
Llama 4 Maverick 17B-128E Instruct FP817B active / 400B totalFP8 ~400GB; IQ1_S 122GB; Q2_K_XL 151GB; Q4_K_XL 243GB1M (430K in practice on 8xH100)Top of the Llama line, shipped with official FP8 weights. Note that its high LMArena placement came from an experimental build that was never released; the public weights scored markedly lower. Do not pick it off a leaderboard.

02 —

Pick a card by scenario

If the VRAM does not add up you get an OOM — each row below budgets weights plus real KV cache

  • Llama 3.1 8B fp16 or 3.2 3B for single-card development

    RTX 3090 24GB$0.193/GPU-hr

    8B in fp16 is 16GB, leaving 8GB for a 32K KV cache — the cheapest card that runs Llama 8B at full precision without compromise.

  • Llama 3.3 70B in AWQ or Q4_K_M for production inference

    A100 PCIE 80GB$0.824/GPU-hr

    Q4 weights are 43GB; only an 80GB card leaves enough KV cache for real concurrency. A 48GB A6000 handles one short-context stream and no more.

  • Llama 3.3 70B at fp16, or LoRA fine-tuning

    A100 SXM4 80GB x2$1.088/GPU-hr

    fp16 weights alone are 141GB, so you need 160GB across two cards — and SXM4's NVLink keeps tensor parallelism off the PCIe bottleneck.

  • Llama 4 Scout at INT4 with long context

    H200 141GB$6.660/GPU-hr

    After 55GB of weights you still have 86GB purely for KV cache — the only single card that takes Scout's context into the hundreds of thousands of tokens.

03 —

Four steps to a running Llama

Worked for Llama 3.3 70B on a single A100 80GB, with both the vLLM and Ollama paths

  1. 01

    Launch an instance on a prebuilt image

    Pick a card from the table above at console.nexgpu.net and boot a prebuilt vLLM or PyTorch image, which skips the whole round of aligning CUDA, driver and PyTorch versions. vLLM, PyTorch and Ollama are all ready-made among the 2,000+ images. Get in over SSH, Jupyter or the web terminal and confirm the GPU is there.

    nvidia-smi && python -c 'import vllm; print(vllm.__version__)'
  2. 02

    Get the weights — meta-llama is a gated org

    Every Llama repo under meta-llama requires you to accept the Llama Community License by hand on the Hugging Face page before your token can pull it. This trips up a lot of people: a 401 in your download script almost always means the agreement was never accepted. 70B is ~141GB at fp16 and ~43GB at Q4, so check your disk first.

    hf auth login && hf download meta-llama/Llama-3.3-70B-Instruct --local-dir /workspace/llama-3.3-70b
  3. 03

    Serve with vLLM — and do not max out the context

    Running 70B on one 80GB card means 4-bit AWQ, which brings weights to about 40GB. The single most important flag is --max-model-len: do not set it to 128K. KV cache scales linearly with context length, so maxing it makes vLLM reserve all remaining VRAM at startup and then OOM. 32K covers almost every workload. You get a standard OpenAI-compatible endpoint.

    vllm serve casperhansen/llama-3.3-70b-instruct-awq --max-model-len 32768 --gpu-memory-utilization 0.92 --port 8000
  4. 04

    Just want to see output? Use Ollama

    If you are not serving concurrency and only want to judge whether the model is good enough, Ollama is far less work than vLLM and pulls GGUF weights automatically. The default llama3.3 tag is the 70B at Q4_K_M, 43GB. To try something cheap first, llama3.1:8b is only 4.9GB and starts instantly on an RTX 3090. Llama 4 Scout needs 8-way tensor parallelism, which fits inside a single 14-GPU node.

    curl -fsSL https://ollama.com/install.sh | sh && ollama run llama3.3:70b-instruct-q4_K_M

What a month of Llama 3.3 70B actually costs

Take an AWQ 4-bit Llama 3.3 70B inference service on an A100 PCIE 80GB at $0.824/GPU-hr. During development at 8 hours a day: 8 x $0.824 = $6.592/day, and 22 working days gives 22 x $6.592 = $145.02. Budget 60GB of storage for the 43GB of weights plus images and datasets: 60 x $0.414 = $24.84/month, for about $169.86 all in. To keep it resident 24/7 as a public service: 730 x $0.824 = $601.52, plus $24.84 of storage, roughly $626.36/month. To go cheaper still, run Llama 3.1 8B resident on an RTX 3090 24GB: 730 x $0.193 = $140.89/month, with room to spare at full fp16 precision. Billing is metered per second and priced per hour, with no minimum and no setup fee — compute billing stops the moment the instance stops, and only storage keeps accruing until you destroy it.

04 —

FAQ

Is Llama still being updated? Which version should I self-host now?

The line stopped at Llama 4 in April 2025; Meta then moved to the closed-weight Muse Spark and the Apache 2.0 Muse Glimmer 30B. But frozen does not mean unusable: Llama 3.1 70B and 3.3 70B remain the most deployed open-weight models in production, with mature, stable toolchains, fine-tuning scripts and quant formats — often less hassle than chasing something new. For a new project go straight to Llama 3.3 70B unless you genuinely need Llama 4's native multimodality or extreme context. NexGPU has cards for both paths: A100 PCIE 80GB at $0.824/GPU-hr for 3.3 70B, H200 141GB for Llama 4 Scout.

How much VRAM does Llama 3.3 70B need? Can a 24GB 4090 run it?

Not on one card. Q4_K_M weights are 43GB, q8_0 is 75GB, fp16 is 141GB — a 24GB 4090 cannot even hold the Q4 weights unless you drop to q2_K at 26GB with CPU offload, by which point the quality loss is not worth it. The honest configuration is a single card of 48GB or more, or two 24GB cards with tensor parallelism. On NexGPU one A100 PCIE 80GB at $0.824/GPU-hr does the job for less than a pair of 4090s ($0.540 x 2 = $1.080/hr), and with no parallelism config to wrestle with.

Meta says Llama 4 Scout runs on one H100 — is the 10M context real?

Fitting the weights and opening the context are two different problems. Scout at INT4 is about 55GB, which does fit inside 80GB, but the remaining VRAM as KV cache only carries you tens of thousands of tokens. vLLM's published configuration is 8xH100 with tensor parallelism for 1M tokens and 8xH200 for 3.6M, and the community reports silent hangs beyond ~3M. Taking Scout to a few hundred thousand tokens on a single card only works on the H200 141GB tier, which is $6.660/GPU-hr on NexGPU, metered per second — run the test, stop the instance, do not buy a card to answer one question.

Are there commercial restrictions on self-hosting Llama?

Yes, and it is not a standard open-source licence. The Llama Community License requires products above 700 million monthly active users to request a separate grant that Meta may refuse at its sole discretion; you must prominently display "Built with Llama"; any model fine-tuned from it must be named starting with "Llama"; and the multimodal variants (Llama 3.2 Vision, Llama 4) explicitly withhold the grant from companies headquartered in the EU. Most internal enterprise deployments never come near these lines. Compliance is your call, compute is ours — NexGPU spans 51 countries and regions across 1,175 verified rentable nodes, so you can pick a region to match your data residency requirements.

Why doesn't Llama 4's MoE save VRAM? Isn't 17B active parameters small?

Active parameters save compute, not memory. Scout computes only 17B per forward pass, but all 109B across its 16 experts must stay resident, because the next token could route to any of them. Maverick is more extreme still: 128 experts, 400B total, about 400GB of FP8 weights. So MoE gives you great throughput numbers and ugly VRAM numbers. To squeeze onto one card you need dynamic low-bit quantization of the Unsloth kind (Scout IQ1_S at 33.8GB, Maverick IQ1_S at 122GB) plus offloading the expert layers to CPU. That is a half-hour experiment on an RTX A6000 48GB at $0.817/GPU-hr on NexGPU.

What card should I rent for LoRA fine-tuning on Llama?

It depends on size. For LoRA on Llama 3.1 8B, 16GB of fp16 weights plus optimizer states and gradients fits a single 24GB card — the RTX 4090 24GB at $0.540/GPU-hr is the value pick, and the RTX 5090 32GB at $0.723/GPU-hr lets you push batch size. LoRA on 70B starts from 141GB of fp16 weights, so two A100 SXM4 80GB ($1.088/GPU-hr) is the entry point, where NVLink pays off clearly for tensor parallelism. NexGPU nodes go up to 14 GPUs and 2,152GB of VRAM, enough for full-parameter fine-tuning, with PyTorch images out of the box, access over SSH, Jupyter, REST API or CLI, and bilingual human support on Telegram with no ticket queue.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.