Document AI / post-OCR understanding

Self-hosting DocFormer: one ICCV paper, and a checkpoint you have to train yourself

DocFormer is Amazon's multi-modal document understanding transformer from ICCV 2021 — 183M base, 536M large, with DocFormerv2 reaching 750M. Its problem was never VRAM. It is that Amazon never published a single line of code or a single checkpoint.

DocFormer · self-hosted

First, what it actually is: DocFormer is not an OCR engine. It consumes OCR output — word-level text tokens plus a bounding box for each word — layers page pixels on top, and does downstream understanding: form key-value extraction, receipt field extraction, document classification, DocVQA. The paper (arXiv:2106.11539) is by Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie and R. Manmatha. Two architectural choices set it apart from the contemporaneous LayoutLM line: the visual branch is a plain ResNet50 rather than a pre-trained object detector, and spatial attention weights are shared across the vision and language streams. Pre-training used three self-supervised tasks — MM-MLM, LTR (Learn To Reconstruct) and TDI (Text Describes Image) — over 5 million pages of IIT-CDIP.

Now the wall everyone hits when they try to deploy it locally: there are no official weights. Search the Hugging Face Hub for docformer and you get an empty list; transformers has no DocFormer model class. DocFormerv2 (AAAI 2024, arXiv:2306.01733) is the same story — a paper and an Amazon Science page, no repo, no checkpoint. The only thing that runs today is an MIT-licensed community reimplementation, github.com/shabie/docformer, maintained by uakarsh and shabie at roughly 290 stars, 42 forks and 280 commits. Its maintainer pre-trained it with MLM on a subset of the IDL dataset and put those weights on Kaggle — note that is not the paper's three-task MM-MLM + LTR + TDI recipe, so numbers will not line up.

So this page covers two real paths. One: if you genuinely want to reproduce DocFormer, here is what fine-tuning costs in VRAM and what the paper's own 17 hours per epoch on an A100 40GB converts to in dollars. Two: if your actual goal is running document parsing on your own hardware, in 2026 the models to look at are document VLMs like DeepSeek-OCR and dots.ocr. Both paths need GPUs, and NexGPU rents everything from a $0.188/GPU-hr Tesla V100 32GB up to a 141GB H200, billed by the second.

01 —

DocFormer variants and what they cost in VRAM

Parameter counts, depths and scores come straight from the papers. VRAM is the AdamW mixed-precision training state at 16 bytes per parameter, excluding activations.

Version	Parameters	VRAM	Context	Notes
DocFormer-base (ICCV 2021)	183M	fp16 weights ~0.37GB / training state ~2.9GB + activations, comfortable on 16GB	512 text tokens	12 layers, hidden 768, 12 heads, plain ResNet50 visual branch. Paper scores: FUNSD 83.34 F1, CORD 96.33 F1, RVL-CDIP 96.17%, Kleister-NDA 85.8 F1. Nearly every reproduction attempt starts here.
DocFormer-large	536M	fp16 weights ~1.1GB / training state ~8.6GB + activations, 24GB minimum, 48GB comfortable	512 text tokens	24 layers, hidden 1024, 16 heads. FUNSD 84.55 F1, CORD 96.99 F1. One counterintuitive real result: on RVL-CDIP classification it scores 95.50%, losing to base at 96.17% — confirm your task actually needs the capacity before paying for it.
DocFormerv2-small (AAAI 2024)	66M	training state ~1.1GB + activations, trivial on any 16GB card	encoder-decoder, 128 image tokens	v2's biggest change is deleting the heavy visual encoder entirely: no Swin, no ViT, just a single linear(conv 2x2) initialised from scratch. The small tier is the cheap way to run ablations.
DocFormerv2-base	232M	training state ~3.7GB + activations, easy on 16-24GB	encoder-decoder	Autoregressive and generative — it emits answers directly instead of doing sequence labelling, so DocVQA, InfoVQA and TabFact need no extra head. Pre-training switches to Token-to-Line and Token-to-Grid on the encoder plus denoising MLM on the decoder.
DocFormerv2-large	750M	fp16 weights ~1.5GB / training state ~12GB + activations, 24-32GB entry, 48GB safer	encoder-decoder	The paper's SOTA tier: DocVQA 87.84 ANLS, FUNSD 88.89 F1, CORD 97.70 F1, TabFact 83.2, InfoVQA 48.8, OCR-VQA 71.5, TextVQA 64.0, ST-VQA 71.8. The authors make a point of beating far larger models — GIT2, PaLI and Flamingo — on scene-text VQA.
Community build: shabie/docformer	matches base at 183M	same as base; a 16GB card runs the forward pass and fine-tuning	max_position_embeddings 512	MIT licensed and the only DocFormer code that actually runs today. Default config: coordinate_size 96, shape_size 96, max_2d_position_embeddings 1000, max_relative_positions 8, image_feature_pool_shape [7,7,256], encoder output (1, 512, 768). The Kaggle weights are MLM-only on an IDL subset, not the paper's checkpoint.

02 —

Pick the card for the job you are actually doing

At this parameter scale the bottleneck is never weight size — it is OCR preprocessing throughput and how many experiments you plan to run.

Reproducing FUNSD / CORD fine-tuning with DocFormer-base 183M
Tesla V100 32GB$0.188/GPU-hr
The paper's own downstream fine-tuning ran on V100 16GB; the 32GB version lets you widen batches and cache OCR features, and it happens to be the cheapest card on our shelf.
Full fine-tune of DocFormer-large 536M or DocFormerv2-large 750M
RTX A6000 48GB$0.817/GPU-hr
750M in AdamW is about 12GB of state; add 512-token attention plus the image branch activations and 48GB means you stop fighting batch size and gradient accumulation.
Pre-training from scratch: MM-MLM + LTR + TDI over IIT-CDIP-scale corpora
A100 PCIE 80GB$0.824/GPU-hr
The paper gives a hard anchor — 17 hours per epoch on an A100 40GB — and 80GB lets you scale the batch. Up to 14 GPUs per node and 2,152GB max node VRAM keeps data parallelism inside one box.
Skipping reproduction and running a document VLM: DeepSeek-OCR 3B or dots.ocr
RTX 4090 24GB$0.540/GPU-hr
3B in bf16 is about 6GB of weights, leaving well over ten GB for vLLM KV cache and high-resolution image tokens — the best value resident inference card at this tier.

03 —

Getting DocFormer running on NexGPU

Four steps from empty instance to a successful forward pass. Step four is the fork: reproduction or production.

01
Boot an instance and install the OCR chain
Pick a PyTorch image from the 2,000+ prebuilt library, then come in over SSH or Jupyter. Key point: DocFormer does no OCR of its own — it needs word-level text plus bounding boxes, so tesseract is a hard dependency, not an option. The community implementation's dataset.create_features calls pytesseract directly.
```
sudo apt update && sudo apt install -y tesseract-ocr && pip install pytesseract transformers torch
```
02
Clone the community implementation (official code was never released)
Do not go hunting for a Hugging Face model id — searching docformer returns nothing. The only usable code is the MIT-licensed shabie/docformer. It is not packaged for pip; you clone it and push its src directory onto sys.path. Kaggle hosts the maintainer's IDL-subset MLM weights plus a FUNSD fine-tuning notebook to start from.
```
git clone https://github.com/shabie/docformer.git
```
03
Run one forward pass and confirm you get (1, 512, 768)
Load BertTokenizerFast from bert-base-uncased, push a document page through create_features, then through ExtractFeatures and DocFormerEncoder. An output shape of (1, 512, 768) means the visual, textual and spatial streams are all wired correctly. This takes seconds on a V100 — do not burn an A100 on a smoke test.
```
v_bar, t_bar, v_bar_s, t_bar_s = feature_extractor(encoding); output = docformer(v_bar, t_bar, v_bar_s, t_bar_s)  # -> (1, 512, 768)
```
04
Fork: keep reproducing, or switch to a document VLM
For paper-level numbers you have to supply the missing MM-MLM + LTR + TDI pre-training yourself, which means multi-GPU A100 time and the bill below. If what you need is invoices, contracts and tables parsed today, swap the image on the same machine and bring up vLLM — DeepSeek-OCR and dots.ocr are both MIT-licensed with vLLM support and both fit on a single 4090. Billing stops when the instance stops, so switching costs almost nothing.
```
vllm serve deepseek-ai/DeepSeek-OCR --trust-remote-code --gpu-memory-utilization 0.85
```

Turning the paper's 17 hours per epoch into a bill

Reproducing FUNSD fine-tuning: FUNSD has only 149 training pages, and DocFormer-base 183M finishes a full run on a Tesla V100 32GB in roughly 1.5 hours, so $0.188 x 1.5 = $0.28. That is the cheapest number on this page — less than the meeting where you decide whether to do it. A full large-tier fine-tuning experiment: RTX A6000 48GB at $0.817 x 12 hours = $9.80. The real expense is pre-training from scratch. The paper's anchor is 17 hours per epoch on an A100 40GB; on a single A100 PCIE 80GB that is 17 x $0.824 = $14.01 per epoch. Move to 8-way data parallel and, at ideal linear scaling, 17 / 8 is about 2.13 hours, so 8 x $0.824 x 2.13 = roughly $14.04 per epoch. The money barely moves; what changes is wall clock, from 17 hours to just over two. Ten epochs lands near $140 and finishes in about 21 hours on 8 cards. Storage is separate: budget 500GB for IIT-CDIP-scale images plus OCR cache and that is $0.414 x 500 = $207/month, about $6.9/day — destroy the volume when you are done, because compute billing stops with the instance but storage runs until deletion. If you take the VLM route instead, an RTX 4090 24GB running a full day is $0.540 x 24 = $12.96, cheaper than one pre-training epoch.

04 —

FAQ

Are there official DocFormer pretrained weights, and where do I download them?

No. Amazon published DocFormer (ICCV 2021) and DocFormerv2 (AAAI 2024) but never released code or checkpoints; searching the Hugging Face Hub for docformer returns an empty list and transformers has no matching model class. What exists is the MIT-licensed shabie/docformer plus the maintainer's Kaggle weights, which were trained with MLM only on an IDL subset. Paper-level quality therefore means pre-training it yourself — exactly what per-second-billed A100 PCIE 80GB at $0.824/GPU-hr is for: run it, stop it, stop paying, no idle cluster to justify.

Does DocFormer handle Chinese or other non-English documents?

Not out of the box. The community implementation uses BertTokenizerFast from bert-base-uncased — a 30,522-entry lowercase English vocabulary that turns CJK text into a wall of [UNK]. Supporting it means a new tokenizer, a rebuilt vocabulary and a full pre-training run, which is effectively training a new model. If your real goal is parsing multilingual invoices, contracts and tables, a multilingual document VLM is the pragmatic route. On NexGPU both are the same machine with a different image: A100 for the former, a single RTX 4090 24GB at $0.540/GPU-hr for the latter.

How much VRAM does fine-tuning DocFormer actually need?

Using 16 bytes per parameter for AdamW mixed precision, the training state is about 2.9GB for base at 183M, 8.6GB for large at 536M and 12GB for DocFormerv2-large at 750M, plus activations for 512-token attention and the visual branch. The paper's own downstream fine-tuning ran on V100 16GB. So: base is more than covered by a Tesla V100 32GB at $0.188/GPU-hr, while large and v2-large are happier on an RTX A6000 48GB at $0.817/GPU-hr. NexGPU has no minimum and no setup fee, so a bad batch size just means swapping cards and starting again.

DocFormer versus LayoutLMv3 or Donut — which should I pick?

If you are writing a paper and need a multi-modal document understanding baseline, DocFormer's three-task pre-training and shared spatial attention are still worth running. For production, check licences first: the LayoutLMv3 model card is CC BY-NC-SA 4.0, explicitly non-commercial, and plenty of teams only discover that at legal review, while Donut, DeepSeek-OCR and dots.ocr are MIT. DocFormer has the longest path to production simply because no official weights exist. Whichever you compare, spinning up several instances on different card types in parallel on NexGPU beats queuing them on one local box.

Is DocFormer still worth deploying in 2026?

As a research architecture, yes — the trade-offs around the ResNet50 visual branch and cross-modal shared spatial attention, and v2's decision to cut all the way down to linear(conv 2x2), make a genuinely instructive evolution to study. As a production system it has two hard blockers: missing official weights, and a 512-token ceiling on an English-only vocabulary. Real document parsing today runs on VLMs — DeepSeek-OCR (3B, MIT, bf16, official vLLM support, 75.7 on olmOCR-bench) and dots.ocr (1.7B LLM backbone, MIT, unified multilingual layout and content). Both fit in one RTX 4090 24GB, and $0.540/GPU-hr stands up a resident service.

How long and how many GPUs does pre-training DocFormer from scratch take?

The paper's anchor is 5 million IIT-CDIP pages at 17 hours per epoch on an A100 40GB. The awkward part is not the GPU — it is running OCR over all 5 million pages first to get the word boxes, which is CPU-bound and deserves its own high-core-count instance rather than tying up an A100. For the GPU phase, 8x A100 PCIE 80GB in data parallel brings each epoch to just over two hours. NexGPU offers up to 14 GPUs per node with 2,152GB max node VRAM, across 1,175 verified rentable nodes and 2,498 GPUs in 51 countries and regions, with no quota request and bilingual Telegram support without a ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.