Skip to main content
NexGPU

Multilingual text model

Actually running BLOOM-176B: the full VRAM math from 329GB bf16 down to a 5.27GB quant

BigScience's 176-billion-parameter multilingual base model, in 72 safetensors shards, with a 2048 context and pure multi-head attention. Here is every tier's real memory footprint, parallelism constraint, and known trap.

Sign up freeView pricing

BLOOM — BigScience Large Open-science Open-access Multilingual Language Model — was finished on 11 July 2022 on 384 A100 80GB GPUs at France's Jean Zay supercomputer, at exactly 176,247,271,424 parameters. The architecture is a Megatron-LM GPT2-lineage decoder-only transformer: 70 layers, 14336 hidden dimension, 112 attention heads, ALiBi relative position bias instead of learned positional embeddings, GeLU activations, and an extra LayerNorm applied to the word embedding layer (StableEmbedding). Pretraining ran on ROOTS: 1.6TB of text, 350B unique tokens, 366B tokens actually seen, sequence length pinned at 2048 throughout. The tokenizer is a multilingual byte-level BPE with 250,680 real tokens, padded to 250,880 in the config so tensor-parallel splits divide evenly.

ROOTS is where BLOOM still holds an edge nobody else from that era can match on paper: Simplified Chinese makes up 16.16% of the corpus, second only to English at 30.03%, ahead of French at 12.9%, Spanish at 10.85%, Portuguese at 4.91% and Arabic at 4.6%. That mix is exactly why BELLE picked bloomz-7b1-mt as its base and fine-tuned it on a million Chinese instructions. The blind spots matter just as much: the 46 natural languages include no Russian, no Japanese, no Korean, no German — the BLOOM+1 ACL paper exists specifically to study bolting those on. What is genuinely scarce is the other end of the list: Wolof, Bambara, Twi, Kikuyu, Fon and Tumbuka still have almost no hundred-billion-scale coverage anywhere else.

The hard part of self-hosting BLOOM is not the download, it is the shape of the memory. The bigscience/bloom repo is 705GB total, weights split across 72 safetensors shards, 329GB of that being pure bf16 parameters. The trap people miss is the KV cache: BLOOM predates GQA and MQA and uses full multi-head attention, so 70 layers times 14336 dimensions works out to roughly 4MB of KV per token — one sequence at the full 2048 context eats 8.2GB, and a batch of 8 burns 66GB on cache alone. Separately, 112 attention heads mean your tensor-parallel degree must divide 112: TP=8 works, TP=6 will not start. And vLLM's BloomForCausalLM supports pipeline parallelism but not LoRA adapters, so parameter-efficient fine-tuning has to go through PEFT with transformers instead.

01 —

Every BLOOM / BLOOMZ variant and what it costs in VRAM

Parameter counts are the official model-card figures; GGUF sizes are measured community files. Context is 2048 across the whole family.

VersionParametersVRAMContextNotes
bigscience/bloom (BLOOM-176B)176,247,271,424 (70 layers / 14336 dim / 112 heads)bf16 ~329GB / Q8_0 191.2GB / Q4_K_M 114.8GB / Q2_K 68.2GB2048The original pretrained base — it continues text, it does not follow instructions. 72 safetensors shards, 705GB repo. The GGUF quants ship split across multiple parts and must be concatenated before loading.
bigscience/bloomz, bloomz-mt (BLOOMZ-176B)176,247,271,424Same class as BLOOM-176B: bf16 ~329GB / Q4_K_M ~115GB2048Instruction-tuned on the xP3 crosslingual task mixture, so it follows instructions zero-shot. bloomz-mt was tuned on xP3mt for non-English prompting — pick it if your prompts are not in English.
bigscience/bloom-7b1, bloomz-7b1, bloomz-7b1-mt7,069,016,064 (30 layers / 4096 dim / 32 heads)bf16 ~14.1GB / Q8_0 8.62GB / Q6_K 6.66GB / Q4_K_M 5.27GB / Q2_K 3.44GB2048The practical tier: bf16 on a single 24GB card, no tricks. Word embeddings alone are 1,027,604,480 parameters (14.5% of the model), which is why the quant files run slightly fatter than a same-size Llama.
bigscience/bloom-3b, bloomz-3b3,002,557,440 (30 layers / 2560 dim / 32 heads)bf16 ~6.0GB / Q4 quant around 2.4GB2048The sweet spot for multilingual ablations and tokenizer studies. KV cache is only ~0.3MB per token, so you can push a very large batch through an evaluation set in one pass.
bigscience/bloom-1b7, bloomz-1b71,722,408,960 (24 layers / 2048 dim / 16 heads)bf16 ~3.4GB2048Word embeddings are 513,802,240 parameters — a full 30% of the model. Full-parameter fine-tuning still fits on one 24GB card, which makes it the right size for low-resource-language continued pretraining trials.
bigscience/bloom-560m, bloomz-560m559,214,592 (24 layers / 1024 dim / 16 heads)bf16 ~1.1GB2048The 250,880 x 1024 embedding table is 256,901,120 parameters — nearly half the entire model. The definitive specimen for studying how a large multilingual vocabulary eats a small model alive.

02 —

Which NexGPU card for which job

Sized against real bf16 weights plus real KV cache, with no optimistic rounding.

  • 560M / 1B7 / 3B for multilingual baselines, tokenizer and vocabulary analysis

    RTX 3090 24GB$0.193/GPU-hr

    Even 3B in bf16 is only 6GB, so 24GB holds the model plus a large evaluation batch, and the lowest rate on the list makes repeat runs effectively free.

  • BLOOMZ-7B1 / 7B1-mt bf16 inference, or LoRA / QLoRA instruction fine-tuning

    RTX 4090 24GB$0.540/GPU-hr

    14.1GB of weights plus roughly 1GB of KV per sequence fits comfortably, and Ada-generation bf16 throughput roughly doubles your fine-tuning iteration speed over the previous generation.

  • Single-GPU BLOOM/BLOOMZ-176B at Q4_K_M (114.8GB) for capability probing or offline batch work

    H200 141GB$6.660/GPU-hr

    141GB swallows the 114.8GB quant on one card, which removes every tensor-parallel communication and tuning headache; the ~26GB left over holds KV for about three full-context sequences.

  • BLOOM/BLOOMZ-176B bf16 full-precision serving or paper reproduction

    A100 SXM4 80GB x 8 (640GB total)$1.088/GPU-hr ($8.704/hr for the node)

    TP=8 divides the 112 attention heads exactly, and after the 329GB of weights land you still have roughly 310GB of KV headroom — enough for dozens of concurrent 2048-token sequences.

03 —

Four steps to a running BLOOM

All on NexGPU's prebuilt vLLM / PyTorch images — SSH in and copy these verbatim.

  1. 01

    Launch, size the volume, pull the weights

    Boot the vLLM prebuilt image and give yourself enough disk up front: 40GB for BLOOMZ-7B1, and for the 176B at 329GB bf16 just provision 400GB. The hf CLI resumes interrupted shards automatically; expect one to two hours to pull all 72 shards of the 176B. Remember that compute stops billing the second the instance stops, but the storage volume keeps billing until you destroy it.

    hf download bigscience/bloomz-7b1 --local-dir /workspace/bloomz-7b1
  2. 02

    Serve it with vLLM on one GPU (the 7B1 route)

    Set --max-model-len 2048 explicitly. BLOOM's config carries no max_position_embeddings, and because ALiBi will not throw on longer sequences, letting vLLM infer the context length silently distorts your KV reservation and quietly degrades output quality — the model saw 2048 and nothing longer during all 350B tokens of pretraining. You get a standard /v1/completions endpoint.

    vllm serve /workspace/bloomz-7b1 --dtype bfloat16 --max-model-len 2048 --gpu-memory-utilization 0.90 --port 8000
  3. 03

    Shard the 176B across eight GPUs with tensor parallelism

    The 112 attention heads must be divisible by your TP degree, which leaves 1/2/4/7/8/14/16/28/56/112 — in practice 8x A100 80GB is the stable landing spot. Loading 329GB takes well over ten minutes on its own, so point --download-dir at a persistent volume and skip the re-download next boot. vLLM's BloomForCausalLM does pipeline parallelism but not LoRA, so do not try to attach adapters here.

    vllm serve bigscience/bloomz --tensor-parallel-size 8 --dtype bfloat16 --max-model-len 2048 --download-dir /workspace/hf
  4. 04

    Or quantize to GGUF and shrink the bill

    llama.cpp's convert_hf_to_gguf.py handles the bloom architecture natively. The 7B1 comes out at 5.27GB for Q4_K_M and 8.62GB for Q8_0, which a single 3090 handles with room to spare. The 176B converts too, but Q4_K_M lands as 114.8GB split into three parts that you must cat back together in order before loading — a step that gets skipped in most write-ups and fails loudly when you do.

    python convert_hf_to_gguf.py /workspace/bloomz-7b1 --outfile bloomz-7b1-f16.gguf --outtype f16 && ./llama-quantize bloomz-7b1-f16.gguf bloomz-7b1-Q4_K_M.gguf Q4_K_M

What all three routes actually cost

Straight off our list rates. Route one, exploring BLOOMZ-7B1 (14.1GB bf16): an RTX 3090 24GB at $0.193/GPU-hr for a solid 24 hours is 24 x $0.193 = $4.63; add a 40GB volume for the weights and your GGUF builds at $0.414/GB-month, which prorates to 40 x $0.414 / 30 = $0.55/day, for $5.18 all in. Route two, the 176B Q4_K_M (114.8GB) on one card: H200 141GB at $6.660/GPU-hr, about 1.5 hours to download and concatenate the parts plus 3 hours of evaluation, so (1.5 + 3) x $6.660 = $29.97. Route three, full bf16 reproduction: A100 SXM4 80GB x 8 = 640GB at 8 x $1.088 = $8.704/hr; two hours to pull and load the weights is $17.41, three hours of benchmarking is $26.11, so a complete 176B bf16 evaluation run comes to $43.52. For contrast, buying the 640GB in route three outright is a six-figure capital expense; here it is forty-three dollars and change. Compute billing stops the second the instance stops, while the storage volume keeps billing until you destroy it — so delete that 400GB volume when the run is done.

04 —

Frequently asked questions

How much VRAM does BLOOM need, and what is the minimum GPU count for the 176B?

329GB of bf16 weights is the hard floor, and 8x A100 80GB (640GB) is the configuration the community has actually validated — TP=8 also divides the 112 attention heads cleanly. To spend less, go GGUF: Q4_K_M at 114.8GB fits one H200 141GB, and Q2_K at 68.2GB fits two A100 80GB. Do not forget the KV cache — BLOOM is pure MHA at 70 layers by 14336 dimensions, roughly 4MB per token, so one full 2048-token sequence claims 8.2GB by itself. On NexGPU the A100 SXM4 80GB is $1.088/GPU-hr and the H200 141GB is $6.660/GPU-hr, with up to 14 GPUs and 2,152GB of VRAM in a single node, available immediately with no quota request.

What is the difference between BLOOM and BLOOMZ, and which should I download?

BLOOM is the raw pretrained base — it continues whatever you give it and does not follow instructions. BLOOMZ was fine-tuned on the xP3 crosslingual task mixture and does follow instructions zero-shot. If your prompts are not in English, take the -mt suffix (tuned on xP3mt), such as bloomz-7b1-mt; for English prompting use bloomz-7b1. The bloomz-*-p3 checkpoints are a research control that the authors themselves describe as worse — do not ship them. BLOOMZ also has a well-known quirk: leave the trailing period off your prompt and it will try to continue your input instead of answering, and its answers skew short by default, so ending with a full stop or explicitly asking it to explain in detail changes the output dramatically. Both lines run on the same vLLM image on NexGPU — swap the repo name and restart.

How good is BLOOM at Chinese, and how much Chinese was actually in the training data?

Simplified Chinese is 16.16% of the 1.6TB ROOTS corpus, second only to English at 30.03% and ahead of French at 12.9% and Spanish at 10.85% — among the highest Chinese proportions in any open model of its generation, which is precisely why BELLE built on bloomz-7b1-mt with a million Chinese instructions. The 250,880-token multilingual BPE vocabulary also segments Chinese far more efficiently than a GPT-2-style 50k vocabulary. Set expectations honestly, though: on today's Chinese benchmarks BLOOM loses to newer models of the same size, and its real value is breadth of language coverage plus a fully traceable corpus. To run your own Chinese comparison, one RTX 3090 24GB at $0.193/GPU-hr on NexGPU runs bloomz-7b1 end to end for less than the price of a coffee.

The context is only 2048 — can I extend it to 4K or 8K?

Architecturally you can try. BLOOM uses ALiBi relative position bias rather than learned positional embeddings, so it has some inherent extrapolation and will not throw past 2048. But all 350B tokens of pretraining happened at 2048, and quality falls off quickly beyond it, so the safe move is to pin --max-model-len to 2048 and do proper continued pretraining if you genuinely need longer. Keep the KV math in view too: at roughly 4MB per token with MHA, a single 8K sequence needs 32.8GB, which will blow up your card well before the weights do. For long-context continued pretraining, NexGPU's RTX A6000 48GB at $0.817/GPU-hr and A100 SXM4 80GB at $1.088/GPU-hr both bill per second and stop when you stop.

Is BLOOM still worth deploying, or has it been superseded?

Honestly: as a general-purpose production model it is legacy now — newer models of the same size score higher, use less VRAM and carry far longer contexts, and no new project should start from BLOOM. The reasons it is still used seriously are specific. First, ROOTS is fully public and source-traceable, making BLOOM one of very few hundred-billion-scale models you can audit end to end for data provenance, memorization research or training-data compliance arguments. Second, its 46 languages include Wolof, Bambara, Twi, Kikuyu and Fon, which almost nothing else at this scale covers. Third, a large body of 2023-era papers uses BLOOM/BLOOMZ as a baseline, and reproducing them requires the original weights. All three are short, bursty jobs — the worst possible reason to buy hardware and the best possible reason to rent by the second on NexGPU, where you spin up, run, and stop.

Does the RAIL licence allow commercial use?

Yes — commercial use, redistribution and derivative models are all permitted, and the persistent claim that BLOOM is non-commercial is simply wrong. The catch is that bigscience-bloom-rail-1.0 is not an OSI open-source licence: Attachment A enumerates prohibited uses, and those restrictions must be passed through unchanged to your downstream users and to any derivative you release. The model card separately marks biomedical, political, legal and finance domains as out of scope, along with scoring individuals for employment, education or credit, and any critical automated decision that has to be correct. Walk Attachment A against your actual use case before you ship rather than stopping at the word commercial. For that round of hands-on validation and compliance review, NexGPU has no quota request, no minimum spend and no setup fee — open an instance and start.

More in Text LLMs

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.