Document OCR model

Surya OCR 2 self-hosted: 650M params won't fill a 24GB card — feeding the vLLM backend is the hard part

Datalab rebuilt Surya from the ground up in v0.20.0, folding OCR, layout analysis and table recognition into a single 650M-parameter VLM. The weights are tiny. The deployment complexity all moved into the inference backend.

Surya · self-hosted

Start with the thing people search wrong: Surya's VRAM question is not about whether the model fits. The model.safetensors in datalab-to/surya-ocr-2 is bf16 and 1.37GB; the GGUF build, surya-2.gguf, is 1.27GB plus a 205MB mmproj projector. What actually decides your card size is vLLM's KV cache and your concurrency. Surya ships with VLLM_GPU_MEMORY_UTILIZATION at 0.85 and VLLM_MAX_MODEL_LEN at 18000, which means that whether you rent 16GB or 80GB, it carves off 85% of it as a cache pool on startup.

Surya 2 is a hard architectural break. The v1 arrangement of FoundationPredictor plus a pile of separate models is gone; layout, recognition and table_rec now share one Qwen3.5-style vision-language model dispatched through SuryaInferenceManager, served by vllm on NVIDIA hardware or llama.cpp on CPU and Apple Silicon. The one component still living in plain PyTorch is the text-line detector (a modified EfficientViT segformer). It depends on no inference backend at all and runs fine without a GPU — a branch that turns out to matter a lot when you pick hardware.

On capability, Surya OCR 2 scores 83.3% on olmOCR-bench, the best result under 3B parameters, and averages an 87.2% pass rate across a 91-language internal benchmark with 38 languages above 90%. It is strong on well-formed modern documents (Base 99.7%, TinyText 93.7%, ArXiv 88.3%, Tables 86.6%) and weak on degraded scans (OldScans sits at 41.8%). The code is Apache 2.0, but the weights ship under a modified AI Pubs OpenRAIL-M licence: free for research, personal use, and companies under $5M in funding or revenue, with a commercial licence from Datalab required past that. Worth reading before it reaches production, not after.

01 —

Which Surya models actually exist right now

The pip package is surya-ocr and v0.22.x is the current line. These are the weights it will pull.

Version	Parameters	VRAM	Context	Notes
datalab-to/surya-ocr-2	650M (Qwen3.5-style VLM)	1.37GB bf16 weights; vLLM claims 0.85 of the card by default	VLLM_MAX_MODEL_LEN 18000; full-page OCR capped at 12,288 tokens	The workhorse. OCR, layout and table recognition all run through this one model. 83.3% on olmOCR-bench.
datalab-to/surya-ocr-2-gguf	650M	surya-2.gguf 1.27GB (F16) + surya-2-mmproj.gguf 205MB	Same model, context governed by your llama.cpp flags	The llama.cpp build. Datalab ships F16 only — there is no official Q4_K_M or Q8_0 quant.
Text-line detector (s3://text_detection/2025_05_07)	Modified EfficientViT segformer, small-model scale	Under 1GB, runs on CPU	96 DPI page images	The only component that needs neither vLLM nor llama.cpp. surya_detect runs standalone — excellent for cropping, line splitting and pre-filtering pages.
datalab-to/surya_layout2 (Fast Layout)	Compact object detector	Negligible; CPU or GPU	96 DPI page images	Added in v0.21.0 as a drop-in alternative to VLM-based layout. Skipping one VLM call makes a visible throughput difference.
datalab-to/chandra-ocr-2 (bigger sibling)	4B class	bf16 wants the A100 80GB / H100 80GB tier	Not published	85.9% on olmOCR-bench, better at math, tables and complex layouts. Costs 6x the parameters and carries a stricter licence threshold ($2M).

02 —

What to rent on NexGPU

The binding constraint isn't weight size — it's vLLM's compute capability 7.5 floor and bf16 needing Ampere or newer.

One card running detect + layout + full-page OCR end to end for evaluation
RTX 3090 24GB$0.193/GPU-hr
Ampere gives you native bf16, and 24GB at 0.85 utilisation leaves enough KV cache for dozens of concurrent requests — the cheapest card here that runs the vLLM backend at all.
Production throughput matching the published benchmark, 128 concurrent requests
RTX 5090 32GB$0.723/GPU-hr
Datalab's published 5.35 pages/s and 12,884 tokens/s were measured on an RTX 5090 with vllm, so this is the one card where you can check your numbers against theirs directly.
Low-concurrency always-on service, or a budget-first long-tail queue
Tesla T4 16GB$0.298/GPU-hr
Turing's compute capability 7.5 just clears vLLM's floor, but it has no bf16 — pass --dtype float16 explicitly or the server exits on startup.
Detection and Fast Layout only, or the llama.cpp GGUF route
Tesla P40 24GB$0.214/GPU-hr
P40 is compute capability 6.1 and V100 is 7.0, both below vLLM's 7.5, so vLLM will not start; but the detector is plain torch and the GGUF runs under llama.cpp, and both are perfectly happy on these cards.

03 —

From bare instance to your first page of HTML

Four steps. Step two comes before the backend on purpose — it validates half the pipeline before you wrestle with Docker.

01
Launch, install, confirm the NVIDIA Container Toolkit is wired up
Surya pulls vllm/vllm-openai:v0.20.1 itself to run the backend, so the host must already have the nvidia runtime registered with Docker. The single most common issue report is unknown or invalid runtime name: nvidia — NexGPU's prebuilt PyTorch and vLLM images have this configured already. Note that Podman is not supported yet.
```
pip install surya-ocr
```
02
Run detection first to validate the backend-free half
Text-line detection is a standalone torch model that needs neither vLLM nor llama.cpp. Get the boxes drawn first and you'll know whether your scan quality is good enough. When recognition looks bad, tuning DETECTOR_TEXT_THRESHOLD (default 0.6) and DETECTOR_BLANK_THRESHOLD (default 0.35) beats swapping models; raise resolution if needed, but keep page width at or below 2048px.
```
surya_detect ./scans --images --output_dir ./out
```
03
Bring up the vLLM backend and run full-page OCR
SURYA_INFERENCE_KEEP_ALIVE defaults to false, which tears the container down after each batch and reloads the weights for the next one. Set it to true for batch work. SURYA_INFERENCE_PARALLEL controls client-side concurrency and usually has headroom up to 64 on a 24GB card. Mind the output schema: in v2, text_lines became blocks carrying HTML.
```
SURYA_INFERENCE_KEEP_ALIVE=true SURYA_INFERENCE_PARALLEL=64 surya_ocr ./scans --output_dir ./out
```
04
Host vLLM yourself and point Surya at it
For multi-node setups, shared cards, or any time you'd rather Surya never touched Docker, run your own OpenAI-compatible server and export SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1. Security note: the container Surya spawns is published with -p {port}:8000 and binds every interface, so firewall a public-IP box or take this self-hosted route instead.
```
vllm serve datalab-to/surya-ocr-2 --max-model-len 18000 --gpu-memory-utilization 0.85 --port 8000
```

What 100,000 scanned pages actually costs

Work it straight from the published benchmark: Surya OCR 2 hits 5.35 pages/s on an RTX 5090 with the vllm backend at 128 concurrency, so 5.35 x 3600 = 19,260 pages/hour. NexGPU's RTX 5090 32GB is $0.723/GPU-hr, so 100,000 / 19,260 is about 5.19 hours, and 5.19 x $0.723 is about $3.75. That is roughly $0.0000375 per page, or about $0.375 per ten thousand pages. Move to an RTX 3090 24GB at $0.193/GPU-hr and the hourly rate drops by more than half; you give up concurrency, but billing is metered per second so running slower costs you nothing extra. The line item that actually blows up your bill is never the model: leave SURYA_INFERENCE_KEEP_ALIVE at its default of false and every batch respawns the vLLM container and reloads the weights, and those idle seconds bill exactly like working ones — set it to true for anything long-running. Stop the instance and compute billing stops immediately; the 1.4GB of weights and your image cache keep accruing storage at $0.414/GB-month (median) until you destroy it, and pulling results out runs at $0.0081/GB egress (median).

04 —

FAQ

How much VRAM does Surya OCR really need? Is 24GB enough?

The weights are 1.37GB, so 24GB is far more than enough to load it. What confuses people is that Surya starts vLLM with VLLM_GPU_MEMORY_UTILIZATION=0.85, which reserves 85% of whatever card you gave it as KV cache — allocated is not the same as required. Size by concurrency instead: 16GB is fine for evaluation, 24GB is the sensible floor for a service handling dozens of parallel requests. A NexGPU RTX 3090 24GB is $0.193/GPU-hr billed per second, so an hour of real load will tell you your concurrency ceiling more reliably than any estimate.

Can I run Surya on an older card like a Tesla V100 or P40?

vLLM requires compute capability 7.5 or higher. V100 is 7.0 and P40 is 6.1, so neither will bring the vLLM backend up. That does not make the cards useless here: the text-line detector is plain PyTorch and Fast Layout is an ordinary object detector, and both run well on older silicon, while full OCR can go through datalab-to/surya-ocr-2-gguf under llama.cpp. On NexGPU the Tesla V100 32GB is $0.188/GPU-hr and the Tesla P40 24GB is $0.214/GPU-hr — the two cheapest cards on the network, and a good fit for a detection and preprocessing pipeline.

What breaks when upgrading from Surya 1 to Surya 2?

Quite a lot. FoundationPredictor is replaced by SuryaInferenceManager, now shared across layout, recognition and table_rec; text_lines in the output became blocks containing HTML; layout results dropped top_k; and table recognition cells no longer carry colspan or rowspan. On top of that, v2 adds a hard runtime dependency — you must provide a vllm or llama.cpp backend. The cheapest way to test the migration without disturbing your existing environment is a clean instance on NexGPU; stop it when you're done and compute billing ends there.

Is Surya free for commercial use?

The repository is Apache 2.0, so the code is unrestricted. The model weights are under a modified AI Pubs OpenRAIL-M licence: free for research, personal use, and startups under $5M in funding or revenue, with a commercial licence from Datalab needed above that. (The bigger Chandra OCR 2 has a lower $2M threshold and forbids competing with their API.) These terms attach to the weights, not to where you run them — self-hosting inference on NexGPU changes nothing about your licence position.

Why does Docker say unknown or invalid runtime name: nvidia?

It's the most frequent failure when Surya spawns its own vLLM container, and it means the host has drivers installed but never registered the NVIDIA Container Toolkit with Docker. Fix it by running nvidia-ctk runtime configure and restarting Docker, or sidestep it entirely by pointing SURYA_INFERENCE_URL at an already-running vLLM server. NexGPU's 2,000+ prebuilt images include PyTorch and vLLM environments with the container runtime already configured, so pip install surya-ocr works on a fresh boot.

Should I use Surya or Chandra, or a larger OCR model?

It depends on your documents. For clean modern PDFs, papers, invoices and tables, Surya OCR 2's 83.3% is plenty, and at 0.65B it delivers several times the throughput of a 4B-class model. If your corpus is mostly degraded scans — Surya scores just 41.8% on OldScans — or dense mathematics, Chandra OCR 2's 85.9% earns those extra parameters. Both paths are open on NexGPU: Surya is happy on a single RTX 3090 24GB at $0.193/GPU-hr, Chandra wants an A100 PCIE 80GB at $0.824/GPU-hr or an H100 SXM 80GB at $3.582/GPU-hr, and you switch between them in the same console with per-second billing throughout.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.