Skip to main content
NexGPU

Vision-language model

Self-host Florence-2: 1.55GB of weights that does detection, OCR and dense captioning

A vision foundation model with 0.77B parameters and 1.55GB of fp16 weights, switching between detection, segmentation, OCR and captioning with a single task token. From $0.193/GPU-hr on an RTX 3090 24GB, metered per second.

Sign up freeView pricing

The name Florence covers two different things. Microsoft's 2021 paper "Florence: A New Foundation Model for Computer Vision" describes a closed vision foundation model most people only ever touched through Azure's image analysis service. What you can actually download, ship and modify is its open successor, Florence-2 (technical report arXiv 2311.06242), MIT licensed, weights on Hugging Face. So when people search for running Florence locally, Florence-2 is almost always what they mean.

The surprising part is how small it is. Florence-2-large is 0.77B parameters and its model.safetensors is 1.55GB; the base version is 0.23B and 463MB. At that size, trained on FLD-5B (5.4 billion annotations over 126 million images), one model handles captioning, detailed captioning, object detection, dense region captioning, phrase grounding, open-vocabulary detection, referring expression segmentation, OCR and OCR with regions. You switch between them not by writing a prompt but by feeding a task token — <OD>, <OCR_WITH_REGION>, <CAPTION_TO_PHRASE_GROUNDING> — as the input.

The architecture is a DaViT vision tower plus a BART-style encoder-decoder text tower, and that shapes how you deploy it. It is not decoder-only, so the usual vLLM fast path does not apply — vLLM lists Florence2ForConditionalGeneration under its bart-plugin rather than as a native architecture. But because the model is tiny, you can spend the entire card on batch size and brute-force the throughput. What costs money here is labelling volume and fine-tuning hours, never VRAM.

01 —

Variants and VRAM

Use the florence-community org — it hosts the official transformers-converted checkpoints

VersionParametersVRAMContextNotes
florence-community/Florence-2-large0.77B1.55GB fp16 weights / under 1GB in bitsandbytes 4-bit4096 (text tower)The default pick. Zero-shot detection, dense region captioning and OCR are all a clear step above base, and it is by far the most downloaded of the four checkpoints.
florence-community/Florence-2-base0.23B463MB fp16 weights4096 (text tower)For chewing through million-image libraries, or squeezing onto CPU and edge devices — the community has already shipped ONNX, CoreML and ExecuTorch ports.
florence-community/Florence-2-large-ft0.77B1.55GB fp16 weights4096 (text tower)Further fine-tuned on a collection of downstream tasks. Better benchmark numbers, weaker zero-shot generalisation than plain large — run both on your own data before choosing.
florence-community/Florence-2-base-ft0.23B463MB fp16 weights4096 (text tower)The fine-tuned small model, for narrow fixed pipelines that only ever need one or two task tokens.
Florence-VL 3B / 8B (jiuhai/florence-vl-*)listed as 5B / 9B on HF~10GB / ~18GB in bf16 (estimated from listed parameter counts)inherited from the base LLMThe follow-on line: Florence-2 used as a vision encoder feeding an LLM for conversational VQA. Go here if you need chat; stay on Florence-2 if you need boxes and coordinates.

02 —

Which card to rent

With 1.55GB of weights, VRAM is not the constraint — throughput and GPU-hours are

  • Single-GPU inference and day-to-day batch labelling

    RTX 3090 24GB$0.193/GPU-hr

    The cheapest Ampere card on the fleet, with full fp16 tensor cores — load 1.55GB of weights and the remaining 20GB is pure batch headroom.

  • High-throughput pipeline captioning a million-image library in one pass

    RTX 4090 24GB$0.540/GPU-hr

    Same 24GB, meaningfully more images per hour — and when billing is per second, the time you save is the money you save.

  • Domain fine-tuning with the vision tower frozen (DocVQA, receipts, UI screenshots)

    A100 PCIE 80GB$0.824/GPU-hr

    The official recipe fits batch size 6 in 40GB with the encoder frozen; 80GB lets you scale the batch up without rewriting your LR schedule.

  • Full fine-tune with the vision tower unfrozen, reproducing the official 8-GPU run

    H100 SXM 80GB ×8$3.582/GPU-hr

    The published batch-64, 7-epoch, 70-minute configuration transplanted as-is — up to 14 GPUs per node, and 8 of them needs no quota request.

03 —

From boot to your first bounding box

Four steps, and trust_remote_code never appears

  1. 01

    Boot a PyTorch instance and install the right stack

    Start from a prebuilt PyTorch image and come in over SSH, Jupyter or the web terminal. Florence-2 now lives in the transformers main branch as Florence2ForConditionalGeneration and Florence2Processor, so trust_remote_code=True is no longer part of the picture.

    pip install -U transformers accelerate timm einops pillow
  2. 02

    Pull the officially converted weights

    Use the florence-community org: base, base-ft, large and large-ft all live there, with configs converted under transformers 4.56.1. The original microsoft/ repo ships the 2024-era modeling_florence2.py custom code, which keeps fighting with newer transformers releases — no reason to walk into that again.

    hf download florence-community/Florence-2-large --local-dir /workspace/florence2-large
  3. 03

    Run your first image with a task token

    The processor resizes every image to a fixed 768×768, the task token decides what the model does, and generation defaults to num_beams=3 with max_new_tokens typically set to 1024. What comes back is a string containing quantised location tokens from <loc_0> to <loc_999> — you must pass it through post_process_generation together with the original image_size to recover pixel coordinates. Parsing the raw string as JSON will give you wrong boxes.

    python -c "from transformers import pipeline; p = pipeline('image-text-to-text', model='florence-community/Florence-2-large', device=0); print(p('demo.jpg', text='<OD>'))"
  4. 04

    Shrink it, or go the other way and scale it

    To go smaller, the transformers docs give a ready bitsandbytes example — BitsAndBytesConfig(load_in_4bit=True) drops the weights under 1GB. To go bigger, do the opposite: spend the whole card on batch size and pad your images toward 768 on the long edge first to avoid pointless rescaling. Stop the instance when the run finishes; compute billing stops immediately, and only storage keeps accruing until you destroy it.

    pip install bitsandbytes

What this actually costs

Start with the expensive end: reproducing Hugging Face's DocVQA fine-tune with the vision tower unfrozen, batch 64, 7 epochs, which they ran on 8 H100s in 70 minutes. On NexGPU that is 8 × $3.582 = $28.656/hr, and 70 minutes means 28.656 × 70 ÷ 60 = $33.43 for the whole run. If budget matters more than wall clock, take their frozen-encoder, batch-6 route instead: a single A100 PCIE 80GB at $0.824/GPU-hr costs 0.824 × 12 = $9.89 even if it drags out to 12 hours. Inference is almost free by comparison — Florence-2-large is 1.55GB of fp16 weights, so an RTX 3090 24GB at $0.193/GPU-hr running a captioning service around the clock is 0.193 × 24 = $4.63/day, and stepping up to an RTX 4090 24GB is still only 0.540 × 24 = $12.96/day. Keep 20GB of dataset plus checkpoints for three days and that is 20 × $0.414 × 3 ÷ 30 = $0.83. Everything is metered per second and priced per hour, with no minimum, no setup fee and no quota request; compute billing stops when the instance stops, while storage runs until you destroy it.

04 —

FAQ

How much VRAM does Florence-2 actually need?

Large is 1.55GB of fp16 weights, base is 463MB, and bitsandbytes 4-bit takes large under 1GB. Around 4GB is comfortable for single-stream inference; what really consumes memory is batch size and beam search. Just don't pick a card on the VRAM number alone: the Pascal-era Tesla P40 24GB is only $0.214 but has no usable fp16 tensor throughput, and the Tesla V100 32GB at $0.188 runs fp16 happily but has no bf16 at all. On NexGPU the safe default for this workload is the RTX 3090 24GB at $0.193/GPU-hr, metered per second, stopped the moment you're done.

Can I serve Florence-2 with vLLM?

Not natively. Florence-2's text tower is an encoder-decoder, and vLLM's supported-models table lists Florence2ForConditionalGeneration under its bart-plugin rather than as a native architecture. In production the practical options are transformers with a large batch, or exporting to ONNX for pipeline inference. NexGPU's 2,000+ prebuilt images include both PyTorch and vLLM, so you can boot one box, benchmark both paths, and spend less than a coffee doing it.

Why won't microsoft/Florence-2-large load on my machine?

Two classic traps. First, flash_attn: the custom code reaches for it at import time and blows up if it never compiled locally — that's the long-running community thread #71. Second, version drift: modeling_florence2.py was written against transformers 4.4x, thread #104 reports it breaking on transformers >= 4.52.1, #115 reports the _supports_sdpa AttributeError on transformers >= 4.54.0, and people are still submitting fixes for the transformers v5 era. Switch to florence-community/Florence-2-large with the native Florence2ForConditionalGeneration and all of it disappears. NexGPU's PyTorch images boot ready to run, which also saves you the afternoon you'd otherwise lose to CUDA build errors.

Is Florence-2's OCR good enough to replace a dedicated OCR engine?

It depends on the images. <OCR> and <OCR_WITH_REGION> are strong on signage, receipts, packaging and UI screenshots — text embedded in a scene — and they hand you the boxes for free. But the vision tower always sees 768×768, so full-page scans and dense small type get mushy once downscaled, which is exactly why the community has been fine-tuning it for OCR since early on. The fastest way to know is to rent an RTX 4090 24GB at $0.540/GPU-hr on NexGPU and run a pass over your own images; that tells you more than any leaderboard.

How many GPUs do I need to fine-tune Florence-2?

Hugging Face's DocVQA walkthrough gives you two reference points: with the vision encoder frozen it fits batch size 6 on a single A100 40GB and even batch 1 on a T4, while the full fine-tune with the encoder unfrozen used 8 H100s at batch 64 for 7 epochs in 70 minutes, at lr 1e-6 with AdamW and a linear scheduler (they warn that a larger learning rate overfits almost immediately). Mapped onto NexGPU: the first is an A100 PCIE 80GB at $0.824/GPU-hr, the second is 8 × H100 SXM 80GB at $3.582/GPU-hr, on nodes that go up to 14 GPUs with no quota request in the way.

Is Florence obsolete? Is there something newer?

Florence-2 is still the current downloadable line — all four checkpoints are maintained and they were converted into the transformers main branch, which is itself a sign the model is being taken seriously. The evolution from here is Florence-VL, which uses Florence-2 as a vision encoder feeding an LLM for conversational VQA (jiuhai/florence-vl-3b and florence-vl-8b on Hugging Face). But if what you want is boxes, coordinates and OCR, nothing MIT-licensed at 0.77B is easier to live with. NexGPU runs both: Florence-2 on an RTX 3090 24GB at $0.193/GPU-hr, and a bf16 Florence-VL 8B on an RTX 4090 24GB at $0.540/GPU-hr or an A10 24GB at $0.414/GPU-hr.

More in Multimodal vision models

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.