Skip to main content
NexGPU

Private LLM Deployment

Open weights, on your own machine

The weights are public; the cards are not. DeepSeek-V4-Flash is 155GB at four-bit, more than one GPU holds. Qwen3.8-27B fits on one once quantised. This page lists the open models whose weights you can actually download, what each one needs, and which node to rent for it.

Sign up freeView pricing
per hour for 27B-class
$0.540
smallest usable V4-Flash quant
103GB
VRAM in one node
2152GB

01 — Decide first

When self-hosting is worth it

Private deployment is not the better option by default. It is a trade: you spend cost and operational attention to buy control. An API has near-zero marginal cost, no ops burden and always-current models. Self-hosting hands you the memory, the concurrency and the upgrades. So be clear about what you are actually buying.

Three cases justify it. First, data that genuinely cannot leave your network — credit risk, medical records, legal contracts, unreleased designs — where compliance is an entry condition rather than a feature. Second, sustained high concurrency: once the API bill sits consistently above a month of dedicated hardware, amortised self-hosting wins. Third, you need to change the model itself: continued pretraining, domain adaptation, deep alignment for structured output, all of which require the weights in hand.

The commoner case is that it is not worth it. A few thousand calls a day, no hard compliance requirement, no inference-ops experience on the team: per-token pricing is almost always cheaper, and the time you save is worth more than the money. The real dividing line is not how large the model is. It is whether you have a reason to keep VRAM occupied continuously.

02 — Where it is required

Workloads that have to stay private

What these share is not model capability. It is that the data cannot leave.

  • Internal knowledge base and document QA

    Index internal policies, product manuals and support history so the model answers only from your own corpus. The whole pipeline runs inside the instance you rented; neither the corpus nor the index passes through a third-party API or into any vendor's training data.

  • Regulated-industry inference

    In finance, healthcare and government work, data leaving the network is itself the incident. Private deployment on a dedicated instance with private networking lets you write down which machine the model runs on, and region-pinned deployment addresses data-residency requirements.

  • Sustained high-concurrency services

    When production traffic stays high, a per-token bill keeps exceeding the cost of one always-on machine. Moving inference onto your own instance turns a variable cost into a fixed one, and removes your exposure to upstream rate limits.

  • Customisation that touches the weights

    Continued pretraining, domain fine-tuning, alignment for structured output: anything that changes the model rather than the prompt needs the weights. This is where an open model's value is actually realised.

03 — The stack

Three routes, chosen by concurrency

The same weights on different inference engines differ sharply in throughput and memory.

  • vLLM / SGLang — production concurrency

    PagedAttention and continuous batching lift memory utilisation substantially, which makes this the default under real load. Both expose an OpenAI-compatible endpoint, so existing code only changes its base_url. The right choice for serving a private model as an internal service.

    vLLM · SGLang · OpenAI-compatible · tensor parallel

  • llama.cpp / GGUF — squeeze the memory

    GGUF quantisation compresses weights to three-to-eight bits, which is what lets an oversized MoE model fit finite memory at all. Community builds are plentiful and CPU/GPU hybrid loading is supported. Throughput is below vLLM, so this suits one person or a small team evaluating.

    llama.cpp · GGUF · 3-8 bit · CPU/GPU hybrid

  • Ollama / Open WebUI — running fastest

    One command pulls a model; add Open WebUI and you have a usable internal chat entry point. Good for validating quality and demoing to stakeholders, then switching to vLLM once the value is confirmed.

    Ollama · Open WebUI · one command

04 — Memory table

What each model needs

Memory has to hold the weights plus the KV cache. Thresholds below are keyed to quantised weight size and matched against the cheapest qualifying model in current inventory; the large ones need a whole multi-GPU node.

Model and quantisationMemory floorCheapest availableNotes
Qwen3.8-27B · four-bit20GBTesla V100$0.188/hr27.78B dense, natively multimodal, Apache 2.0. Fits a single card once quantised, the easiest tier of private deployment to land.
DeepSeek-V4-Flash · three-bit IQ3_XXS110GBH200$6.660/hrRoughly 103GB of weights. A 284B-total, 13B-activated MoE; runs under aggressive compression with some quality loss. One large-memory card or a multi-GPU node both carry it.
DeepSeek-V4-Flash · four-bit Q4_K_XL162GBB200$10.797/hrRoughly 155GB, the most common production pick. Needs 162GB of usable memory, on either a single large-memory card or a multi-GPU node.
DeepSeek-V4-Pro / Qwen3.8-Max400GBMulti-GPU node1.6T and 2.4T-class MoE. No official GGUF; community builds sit near 400GB even at Q2_K, so self-hosting is impractical. Rent a multi-GPU node by the hour to evaluate instead.

"Cheapest available" is derived from live inventory — the lowest per-GPU rate among models that clear the VRAM bar — and moves with the market. Multi-GPU nodes rent whole.

05 — Getting started

Your first private model in three moves

  • 01

    Pick a node from the memory floor

    Use the table above to settle on a quantisation tier and its memory floor, then choose the matching single- or multi-GPU node. Multi-GPU nodes rent whole and pool their VRAM, so you are not wiring up interconnect yourself.

  • 02

    Start the inference engine

    Boot vLLM or Ollama from a pre-built image, or pull your own Docker image. Weights can be fetched into the instance from a public repository or from your own private registry.

  • 03

    Connect and load-test

    Point the OpenAI-compatible endpoint at the instance port and change one line of base_url. Run a load test at real concurrency to confirm throughput and memory headroom before deciding between hourly and reserved.

06 — FAQ

About private LLM deployment

How much VRAM does DeepSeek V4 need to self-host?

It depends on the quantisation. For DeepSeek-V4-Flash (284B total / 13B activated), community GGUF builds land at roughly 103GB of weights for three-bit IQ3_XXS with a recommended 110GB floor; about 155GB for four-bit Q4_K_XL with a 162GB floor; and about 162GB for the near-lossless Q8_K_XL with a 169GB floor. The V4-Pro flagship (1.6T total / 49B activated) has no official GGUF, and community builds remain near 400GB even at Q2_K, which makes local hosting impractical. These are third-party measurements, and real usage adds KV cache on top, growing with context length.

Can Qwen3.8 be self-hosted, and does one card do it?

Yes. Qwen3.8-27B was released under Apache 2.0 on 14 August 2026 (27.78B dense, natively multimodal, 262K context extensible to 1M) and runs on a single card once quantised. Weights for the larger Qwen3.8-Max (2.4T total / roughly 95B activated) were published on 12 August, but standing up inference for a 2.4T model is a serious undertaking; renting a multi-GPU node by the hour to evaluate is usually the better route.

When does self-hosting beat calling an API on cost?

The dividing line is how long VRAM stays occupied. Hourly rental bills only while the instance runs, so for evaluation, batch jobs and intermittent use it is almost always cheaper. Reserved or monthly capacity only overtakes per-token pricing once a production service needs VRAM resident around the clock at consistently high volume. The one factor cost cannot settle is compliance: when data may not leave the network, this stops being an arithmetic question.

Are my data and weights safe when self-hosting here?

Instances are dedicated to you; weights and data written while an instance runs live on the node you rented and are released when you destroy it. Enterprise plans add data isolation, private networking and region-pinned deployment. One distinction worth being precise about: private deployment on public cloud means your model and data do not pass through a third-party inference API, not that the facility is physically isolated. If your requirement is the latter, the dedicated-cluster route is the one to discuss. The Terms of Service and Privacy Policy govern the specifics.

Can every open-source model be self-hosted?

No, and the word gets used loosely. Some models only ever open an API and never publish downloadable weights. DeepSeek-V4-Flash-Vision-Exp, for instance, went live on the API on 21 August 2026 with no published parameter count and no weight download, so it cannot be deployed privately. Check three things when choosing: whether the weights are genuinely downloadable, whether the licence permits commercial use (MIT and Apache 2.0 are the most permissive), and whether any geographic restriction applies.

Related solutions

Other ways to use it

Same compute network — swap the image and it becomes a different production line.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.