Skip to main content
NexGPU

Agent & Retrieval Framework

Self-hosting LlamaIndex: the framework costs no VRAM — the three models behind it do

llama-index-core 0.14.24 with llama-index-workflows 2.23.3 is the current line. Put the embedding model, the reranker and the LLM on one card. NexGPU starts at $0.193/GPU-hr, billed per second.

Sign up freeView pricing

First, the framing. LlamaIndex is not a set of weights — it is the document-agent and retrieval framework maintained by run-llama, MIT licensed, 51.8k stars and 8.0k forks on GitHub, with 300+ integration packages on LlamaHub. So the answer to "how much VRAM does LlamaIndex need" is never about the framework itself: it orchestrates somebody else's models. One thing to note before you start: the docs moved wholesale from docs.llamaindex.ai to developers.llamaindex.ai with a 301, and the path structure changed too, so links in older tutorials land somewhere new.

The current package layout surprises people. Workflows has been fully extracted from core onto its own version line, llama-index-workflows 2.23.3, imported as `from workflows import Workflow, step` — `llama_index.core.workflow` survives only as a stability shim. The old workflows-py repo folded into the run-llama/llama-agents monorepo under the LlamaAgents brand, and service deployment now runs through llama-agents-server 0.7.1 plus the llamactl CLI. The older llama-deploy still pins llama-index-core below 0.14.0, which collides head-on with today's 0.14.24 — do not start a new project there.

Practically, self-hosting LlamaIndex means replacing exactly three defaults. In llama_index.core.settings, the `Settings.llm` getter falls through to `resolve_llm("default")`, which constructs an `OpenAI()` and validates the key, raising "Could not load OpenAI model" when it fails. `Settings.embed_model` defaults to BAAI/bge-small-en, an English-only model. `Settings.context_window` defaults to the constant 3900. Point those at a local vLLM endpoint, bge-m3, and your model's real context length, and LlamaIndex is fully offline — after that, all that is left is finding a card with enough VRAM for those three models.

01 —

The version lines still under maintenance

Six packages, six independent version lines — mismatch them and they lock each other

VersionParametersVRAMContextNotes
llama-index 0.14.24Meta-package, only 4 direct deps0GB framework; calls the OpenAI API by defaultSettings.context_window defaults to 3900The `pip install llama-index` starter bundle. It drags in llama-index-llms-openai 0.7.10 and llama-index-embeddings-openai — which is exactly what you do not want in an air-gapped build.
llama-index-core 0.14.24Pure core, requires Python >=3.10, <4.00GB, every model is externalDetermined by the LLM metadata you injectThe real starting point for self-hosting. Settings, node parsers, retrievers, FunctionAgent and AgentWorkflow all live here, with no cloud LLM dependency.
llama-index-workflows 2.23.3Event-driven runtime, installable standalone0GB, pure orchestrationState persisted keyed by run id + namespaceNow on its own version line. `list[E]` fan-out/fan-in joins, `@catch_error` to catch exhausted retries, and snapshot tick replay so a crashed agent resumes instead of restarting.
llama-agents-server 0.7.1 / llama-agents-client 0.3.12Starlette + uvicorn service shell0GB, separate from the inference processStreaming, human-in-the-loop, persisted runsWraps any Workflow as a REST service, or mounts inside a FastAPI app you already have. Pair it with llamactl for init / serve / deployments create.
LlamaIndex.TS 0.12.1 (npm)TypeScript implementation, also MIT0GB, usually front-end orchestration onlyFollows whatever model you attachThe Node and Edge counterpart, versioned entirely separately from Python and narrower in coverage. Keep heavy retrieval logic on the Python side.
llama-deploy 0.9.2 (superseded)Pins llama-index-core <0.14.00GBOld Workflow contract onlyIts core ceiling fights today's 0.14.24 and installing it silently downgrades core. Migrate to llama-agents-server plus llamactl — that is the path the project is pushing.

02 —

Pick the card by what you actually run

VRAM goes to the embedding model, the reranker and the LLM — the framework takes none

  • Indexing only: batch embedding plus reranking, no generation

    RTX 3090 24GB$0.193/GPU-hr

    bge-m3 at fp16 is ~1.2GB and bge-reranker-v2-m3 (568M params) ~1.1GB, leaving 20+GB to spend entirely on embed_batch_size — and this is the cheapest per-hour rate on the fleet.

  • Full local stack on one card: Qwen3-8B generation + bge-m3 retrieval + rerank

    RTX 5090 32GB$0.723/GPU-hr

    Qwen3-8B is really 8.19B params, ~16.4GB in bf16; add 2.3GB of retrieval models and you are at 19GB before any KV cache. A 24GB card leaves too little headroom for long contexts; 32GB breathes.

  • FunctionAgent that reliably calls tools: Qwen3-32B AWQ-INT4 at 32K+ context

    RTX A6000 48GB$0.817/GPU-hr

    32.76B params quantised to INT4 lands around 20GB of weights, and 48GB feeds a long KV cache plus several concurrent runs. Tool-call reliability is a clear step above 8B-class models, which is what keeps multi-agent handoffs from stalling.

  • AgentWorkflow handoffs with a 32B-class model at full bf16

    A100 SXM4 80GB$1.088/GPU-hr

    Qwen3-32B is 65.5GB of bf16 weights alone, so only 80GB-class cards hold it. Handoffs grow the context with every hop, and an OOM mid-run breaks the workflow's persisted state.

03 —

Four steps from a blank instance to a local RAG agent behind REST

Everything below runs on one NexGPU box

  1. 01

    Boot the instance and install the minimum viable package set

    Pick a PyTorch or vLLM prebuilt image in the NexGPU console and SSH in. Skip the llama-index meta-package — it pulls the OpenAI LLM and embedding deps along with it. Start from core and add integrations deliberately; uv keeps the 300-package universe from silently downgrading core on you.

    uv pip install "llama-index-core>=0.14.24" llama-index-llms-openai-like llama-index-embeddings-huggingface llama-index-postprocessor-flag-embedding-reranker llama-index-vector-stores-qdrant
  2. 02

    Bring up the vLLM OpenAI-compatible endpoint first

    LlamaIndex does not do inference; it is an HTTP client. Start the model first, and be sure to enable tool-call parsing — without it FunctionAgent never receives structured tool_calls and degrades into asking the model to emit JSON by hand. Cap gpu-memory-utilization, because bge-m3 and the reranker still need room on the same card.

    vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --max-model-len 32768 --gpu-memory-utilization 0.72 --enable-auto-tool-choice --tool-call-parser hermes --port 8000
  3. 03

    Override the three defaults that will otherwise bite you

    OpenAILike ships with is_chat_model=False and is_function_calling_model=False, and its context_window inherits 3900. Leave those alone and you get an agent that never calls a tool and truncates a 32K context down to 3900 tokens. On the embedding side, device is auto-inferred to CUDA, but DEFAULT_EMBED_BATCH_SIZE is 10 — leave it and the GPU idles.

    from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.llm = OpenAILike(model="qwen3-8b", api_base="http://127.0.0.1:8000/v1", api_key="EMPTY", context_window=32768, is_chat_model=True, is_function_calling_model=True)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-m3", device="cuda", embed_batch_size=64)
  4. 04

    Build the index, start the agent, wrap it as a service

    Raise similarity_top_k when you build the query engine — the default is 2, which returns two nodes and misses most of a long document. Memory is the current memory class; ChatMemoryBuffer, ChatSummaryMemoryBuffer, VectorMemory and SimpleComposableMemory are all marked deprecated. Finally, WorkflowServer exposes the agent as a REST service with streaming and human-in-the-loop, and other services call it with llama-agents-client.

    from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.memory import Memory
from llama_agents.server import WorkflowServer

query_tool = index.as_query_engine(similarity_top_k=8, node_postprocessors=[reranker])
agent = FunctionAgent(tools=[...], llm=Settings.llm, system_prompt="...")
memory = Memory.from_defaults(session_id="u-1", token_limit=40000)

server = WorkflowServer()
server.add_workflow("rag", agent)

What one full self-hosting evaluation actually costs

Concretely. Take a two-million-character internal document corpus. Step one: index it with bge-m3 on an RTX 3090 24GB ($0.193/GPU-hr) at embed_batch_size 64, roughly 3 hours for the full pass — 3 x 0.193 = $0.579. Step two: move to an RTX 5090 32GB ($0.723/GPU-hr) running Qwen3-8B for an internal trial, 8 hours a day for 5 days, 40 hours total — 40 x 0.723 = $28.92. The index, weights and vector store together occupy 20GB of persistent storage; at the $0.414/GB-month median that is 20 x 0.414 x 7 / 30 = $1.93 for the week. Add them up: 0.579 + 28.92 + 1.93 = about $31.43 to settle the self-hosting question in a week. Billing is metered per second and priced per hour, with no minimum, no setup fee and no quota request. Compute billing stops the moment the instance stops; storage keeps accruing until you destroy it — so clean up when the evaluation is done.

04 —

FAQ

How much VRAM does self-hosting LlamaIndex actually need?

The framework is a pure Python orchestration layer: 0GB. What consumes VRAM is the trio behind it — bge-m3 at fp16 ~1.2GB, bge-reranker-v2-m3 at fp16 ~1.1GB, and whichever generator you pick: Qwen3-8B is 16.4GB of bf16 weights, Qwen3-32B is 65.5GB, and that 32B drops to roughly 20GB at INT4. So the question always decomposes into "which LLM am I serving". On NexGPU you can index on an RTX 3090 24GB at $0.193/GPU-hr, then move to a 5090 32GB, an A6000 48GB or an A100 80GB depending on the generator — per-second billing, and switching cards does not mean setting up a new account.

Why do I get "Could not load OpenAI model" when I am trying to use a local model?

Because the `Settings.llm` getter calls `resolve_llm("default")` when you never assigned one, and that branch hard-codes constructing an `OpenAI()` and validating the api_key, raising that asterisk-wrapped error when validation fails. `Settings.embed_model` similarly resolves to BAAI/bge-small-en. The fix is not a fake key — it is assigning Settings.llm and Settings.embed_model explicitly before any index operation. NexGPU's vLLM prebuilt image serves an OpenAI-compatible port straight out of boot: point api_base at it and set api_key to EMPTY.

My FunctionAgent refuses to call tools against vLLM, and the context keeps getting truncated. Why?

This is the classic OpenAILike trap: is_chat_model and is_function_calling_model both default to False, and context_window inherits the core constant 3900. The first makes LlamaIndex assume the model cannot do function calling, so it never sends the tools parameter; the second makes the prompt helper cut your 32K or 128K context down to 3900 tokens. Set all three explicitly in the constructor. The vLLM side also needs --enable-auto-tool-choice with the matching --tool-call-parser — both ends must be on. On a NexGPU vLLM image that is a one-line change to the launch args, so you can verify it in minutes.

Why is retrieval quality worse than expected on the same corpus?

Usually three defaults at once. First, HuggingFaceEmbedding defaults to BAAI/bge-small-en, an English-only model that collapses on non-English corpora; bge-m3 or Qwen3-Embedding-0.6B (596M params, ~1.2GB bf16) changes the picture immediately. Second, DEFAULT_SIMILARITY_TOP_K is 2 — two nodes will miss most of a long document, so push it to 8 and let a reranker narrow it back down. Third, DEFAULT_CHUNK_SIZE is 1024 tokens with only 20 tokens of overlap, which rarely suits dense technical text. These are all configuration, not architecture: an RTX 3090 24GB for under a dollar lets you A/B several parameter sets in an afternoon.

LlamaIndex or LangChain?

The split is real. LlamaIndex is centred on documents — parsing, chunking, indexing, retrieval and reranking carry the thickest abstractions, and the project positions itself as a document agent and OCR platform, with LlamaParse covering 130+ formats. On orchestration, Workflows is event-driven: steps are async functions that consume and emit events, so branching, looping and parallelism are plain Python with no graph DSL to learn, plus built-in persisted state and snapshot replay. If your problem is a pile of unstructured documents plus long-running processes that must resume, LlamaIndex fits better. Neither framework consumes VRAM — the inference box behind it sets your cost. NexGPU spans 51 countries and regions, 1,175 verified rentable nodes and 75 GPU models, so you can run both stacks side by side on the same corpus.

Can I keep using ServiceContext, ChatMemoryBuffer and llama-deploy in an older project?

All three are due for migration. ServiceContext has been replaced by Settings and there is a dedicated migration guide. ChatMemoryBuffer, ChatSummaryMemoryBuffer, VectorMemory and SimpleComposableMemory are all deprecated in favour of `Memory.from_defaults(token_limit=...)`, which handles the short-term/long-term token ratio and flush size for you. And llama-deploy pins llama-index-core below 0.14.0, so installing it downgrades your core — move to llama-agents-server plus llamactl. The risk in any migration is wrecking a working local environment, so spin up a clean per-second NexGPU instance, run the old and new stacks in parallel, and shut it down when you have your answer — usually for less than the price of a coffee.

More in AI agents and workflows

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.