Skip to main content
NexGPU

Speech Recognition / ASR Toolkit

Self-host WeNet: the VRAM, the card, the training bill

WeNet is not a model. It is a full Chinese-first ASR pipeline that runs from data prep through training to a Triton endpoint. Here are the real checkpoint sizes, per-stream VRAM, published benchmarks, and the NexGPU card that fits each one.

Sign up freeView pricing

WeNet came out of the NPU ASLP lab and Mobvoi, formalised in the Interspeech 2021 WeNet paper and the 2022 WeNet 2.0 follow-up. Its thesis is U2/U2++: one set of weights serving both streaming and non-streaming, CTC producing hypotheses that the attention decoder then rescores in a second pass, and a training run that exports straight to LibTorch, ONNX Runtime, OpenVINO, TensorRT or Triton. Apache-2.0, 5,200+ stars, still receiving commits as of June 2026.

One thing to know before you pin a version: v3.1.0 is the last tagged release. It brought FSDP support, flash attention fixes, a Python 3.10 floor, and a recommendation to use step_save instead of epoch_save. Everything after that lives on main — the command-line tool, SenseVoice, FireRedASR, whisper-large-v3-turbo, the Chinese punctuation model. That is why the official README tells you to install from git rather than from a version number.

Which means "how much VRAM does WeNet need" is really a question about which checkpoint you load. The AISHELL baseline is 46M parameters in a 181MB package; FireRedASR-AED-L is 1.1B parameters in a 4.6GB package. More than twenty times apart. ASR models sit two orders of magnitude below LLMs, so at inference the VRAM goes to concurrency and beam width, not weights. What genuinely needs multiple cards is training from scratch — the official WenetSpeech recipe records 8x RTX 3090 for 4.6 days on 10,000 hours.

01 —

Checkpoints WeNet can load, with real sizes

Package sizes are the actual file sizes in the official ModelScope distribution; VRAM figures are single-stream inference footprints

VersionParametersVRAMContextNotes
wenetspeech u2++ conformer (CLI: -m wenetspeech)~120M (12-layer, 512-dim Conformer + 3+3 BiTransformer)470MB package, ~2GB peak FP32 single stream640ms streaming chunks or full-utterance offline, same weightsThe general-purpose Chinese workhorse. Official recipe reaches 5.87 / 8.87 / 11.79 CER on Dev / Test_Net / Test_Meeting
aishell u2++ conformer~46M (12-layer, 256-dim Conformer, 4 heads, 2048 FFN)181MB package, ~1.5GB peak FP32 single streamDynamic-chunk training, chunk size chosen at deploy timeThe reproduction and teaching baseline, trainable on a single card. Hits 4.11 CER on AISHELL-1 with HLG (k2 LM) + attention rescoring + LFMMI
paraformer (CLI: -m paraformer)~200M class, non-autoregressive (Paraformer-Large)909MB package, ~3GB peak FP32 single streamWhole utterance in one forward pass, no autoregressive decodeChinese and English. The lowest latency-variance option for an always-on service, because long utterances do not drag out a decoder loop
sensevoice_small (CLI: -m sensevoice_small)Same class as Whisper-Small (~200M)868MB package, ~3GB peak FP32 single streamNon-autoregressive; one pass yields text, language ID, emotion and audio eventsMandarin, Cantonese, English, Japanese, Korean. Reported at over 5x faster than Whisper-Small at comparable size. Note the weights ship under the FunASR model licence, not Apache-2.0
firered (FireRedASR-AED-L)1.1B4.6GB package (~4.3GB FP32 weights), 8GB+ peak single stream60 second cap per utterance (the LLM variant caps at 30s)Among the strongest open Chinese CERs available: 0.55% on aishell1, 2.52% on aishell2, 4.88% on WenetSpeech. Keep utterance lengths similar within a batch
whisper-large-v3 / whisper-large-v3-turbo1.55B / 809M2.43GB / 1.26GB packages, ~10GB / ~6GB peak FP16 single stream30 second sliding windowsMultilingual fallback. Gotcha: the small Whisper keys in the hub are spelled whiper-tiny / whiper-base / whiper-small / whiper-medium — only the two large-v3 entries are spelled correctly

02 —

Pick the card by what you are doing

WeNet spends VRAM on concurrency and training, not on weights — size the card accordingly

  • Always-on U2++ Conformer or Paraformer transcription service

    RTX 3090 24GB$0.193/GPU-hour

    Weights are under 1GB, so all 24GB goes to batched concurrency and CTC prefix beams — and this card is both cheaper and 8GB larger than a T4

  • Reproducing the official Triton + ONNX FP16 GPU benchmark

    Tesla T4 16GB$0.298/GPU-hour

    The published RTF of 0.0009–0.0011 and offline throughput of 88–228 requests/second in runtime/gpu were measured on a T4, so match the card if you want comparable numbers

  • High-accuracy bulk transcription with FireRedASR-AED-L or Whisper large-v3

    RTX 4090 24GB$0.540/GPU-hour

    FP32 weights alone are 4.3GB at 1.1B parameters; 24GB is what lets you push the batch size high enough to make autoregressive decoding worth it

  • Training U2++ Conformer from scratch on 10,000 hours of WenetSpeech

    RTX 3090 24GB x 8$0.193/GPU-hour ($1.544/hour for all 8)

    That is literally the official recipe's own configuration: 8x3090, dynamic batch 36000, acc_grad 4. Nodes go up to 14 GPUs if you want to finish sooner

03 —

Four steps to a running WeNet

Every command below is taken from the official README and example recipes, unmodified

  1. 01

    Boot a PyTorch instance and install WeNet

    Start from a prebuilt PyTorch image, create a conda env on Python 3.10 (v3.1.0 raised the floor), install the recommended torch 2.2.2+cu121, and add sox via conda install conda-forge::sox. Install from git, not a version pin — the CLI, SenseVoice and FireRedASR only exist on main.

    pip install git+https://github.com/wenet-e2e/wenet.git
  2. 02

    Prove the recognition path with one command

    The CLI pulls weights from ModelScope and caches them locally. The first paraformer run downloads 909MB; firered downloads 4.6GB. Swap models by changing -m: wenetspeech, paraformer, firered, sensevoice_small, whisper-large-v3-turbo. Chinese punctuation is a separate 1.07GB punc model.

    wenet -m paraformer audio.wav
  3. 03

    Train multi-GPU or fine-tune on your own data

    The recipes launch DDP through torchrun, and train_engine switches between torch_ddp, deepspeed and fsdp. For anything over a few hundred hours, shard your data into tar archives first and set data_type to shard — feeding a raw list will saturate I/O long before it saturates the GPUs, and that is the single most common way self-hosted training runs stall.

    torchrun --nnodes=1 --nproc_per_node=8 wenet/bin/train.py --train_engine torch_ddp --config conf/train_u2++_conformer.yaml --data_type shard --train_data data/train/data.list --cv_data data/dev/data.list --model_dir exp/u2pp_conformer --ddp.dist_backend nccl --num_workers 8 --prefetch 100 --pin_memory
  4. 04

    Export ONNX FP16 and serve it behind Triton

    The GPU path is runtime/gpu's Triton setup (the reference image is Triton 22.03, gRPC on 8001 and HTTP on 8000). The export script takes --fp16 and --streaming, with decoding_chunk_size defaulting to 16, i.e. 640ms per chunk. Measured CER difference between FP16 and FP32 is under 0.1%. Run the container with --shm-size=1g and --ulimit memlock=-1.

    python wenet/bin/export_onnx_gpu.py --config exp/u2pp_conformer/train.yaml --checkpoint exp/u2pp_conformer/final.pt --output_onnx_dir onnx_model --fp16 --streaming --decoding_chunk_size 16

What it actually costs

Inference first. WeNet's own Triton benchmark on a Tesla T4, running the AIShell-2 U2++ Conformer offline model, records an RTF of 0.0009–0.0011 and 88–228 requests/second. At RTF 0.001, one T4-hour transcribes roughly 1,000 hours of audio: $0.298/GPU-hour divided by 1,000 audio-hours is about $0.30 per thousand hours. Even if your real-world audio is longer and messier and you discount that by 3x, a thousand hours still costs a little over a dollar. Keeping an RTX 3090 up for a full month instead is 0.193 x 730 = about $140.9. Now training. The official WenetSpeech run — 8x RTX 3090, dynamic batch 36000, acc_grad 4, 130k steps — took 4.6 days. That is 4.6 x 24 = 110.4 hours, and 8 x $0.193 = $1.544/hour, so 110.4 x 1.544 = about $170.5 for a 10,000-hour Chinese ASR foundation model. The variant that folds in AISHELL4 and runs 8.5 days works out to 8.5 x 24 x 1.544 = about $315. In both cases compute is metered per second and billed per hour, and the meter stops when the instance stops. Storage is separate at a $0.414/GB-month median, egress at a $0.0081/GB median. No minimum term, no setup fee, no quota request.

04 —

Frequently asked questions

How much VRAM does self-hosting WeNet actually need?

It depends entirely on the checkpoint. The AISHELL baseline is 46M parameters in a 181MB package and runs in about 1.5GB FP32 per stream. The WenetSpeech U2++ Conformer is a 470MB package in the 2GB range. Paraformer and SenseVoice-Small sit around 3GB. FireRedASR-AED-L, at 1.1B parameters and a 4.6GB package, starts at 8GB, and Whisper large-v3 with FP16 beam decoding wants 10GB of headroom. Weights are never the constraint — batch concurrency is. The cheapest way to find your real number is to boot an RTX 3090 24GB on NexGPU at $0.193/GPU-hour, push concurrency until you hit your target QPS, and only then decide whether you need a bigger card.

Is WeNet still maintained, and what is the current version?

v3.1.0 is the last tagged release: FSDP support, flash attention fixes, a Python 3.10 floor, step_save recommended over epoch_save. But the repo itself is very much alive — commits as recently as June 2026, Apache-2.0, 5,200+ stars, 1,100+ forks. Everything added since v3.1.0 lives on main: the CLI, SenseVoice, FireRedASR, whisper-large-v3-turbo, the punctuation model. Hence the git-based install in the README. The same wenet-e2e org also ships west (an LLM speech toolkit with TouchASU / TouchTTS / TouchChat / TouchOmni), wespeaker, wekws, WeTextProcessing and wetts, so the whole speech stack is available under one roof. If you want to evaluate WeNet mainline and west side by side, spin up two NexGPU instances and run them in parallel — per-second billing means neither one waits on the other.

Should I pick WeNet, FunASR, or Whisper?

That framing is wrong, because WeNet's CLI already loads Paraformer, SenseVoice-Small, FireRedASR and the whole Whisper family — the FunASR line of weights is reachable from inside WeNet. The real distinction is engineering: WeNet gives you U2/U2++ unified streaming and non-streaming modelling, CTC plus attention-rescoring two-pass decoding, WFST context biasing, and export paths to LibTorch, ONNX Runtime, OpenVINO, TensorRT and Triton. It is built to ship, not to top a leaderboard. The only honest comparison is running each checkpoint against your own test set on the same card. NexGPU carries 2,000+ prebuilt images including PyTorch, vLLM and Whisper ASR, so that comparison starts the moment the instance boots.

How long does training a Chinese ASR model from scratch take, and on how many GPUs?

The official WenetSpeech recipe publishes hard numbers: 8x RTX 3090, dynamic batch 36000, acc_grad 4, 130k steps, 4.6 days. An earlier Conformer variant used 24x V100 for 26 epochs, and the run that mixes in AISHELL4 took 8.5 days on 8x3090. If you are only fine-tuning on a few hundred hours of your own audio, 4 to 8 3090s for a day or two is usually enough. NexGPU nodes go up to 14 GPUs with a maximum node VRAM of 2,152GB, RTX 3090 is $0.193/GPU-hour and Tesla V100 32GB is $0.188/GPU-hour — eight cards running flat out for a full day is about $37, and the meter stops the moment training finishes.

wenet -m whisper-small says the model does not exist. Did I install it wrong?

No, that one is a typo in the hub. The small Whisper keys are spelled whiper-tiny, whiper-base, whiper-small and whiper-medium — missing an s — while whisper-large-v3 and whisper-large-v3-turbo are spelled correctly. Copy the exact strings from wenet/cli/hub.py and the download works. Separately, weights are distributed through ModelScope's OSS API, so the first pull of something like the 4.6GB firered package can be slow from the wrong part of the world. NexGPU has 1,175 verified rentable nodes and 2,498 GPUs across 51 countries and regions — pick one close to your data and your users.

Streaming or non-streaming for a production service, and what latency should I expect?

That choice is exactly what U2++ was designed to defer: dynamic chunks during training, chunk size decided at deployment. The official Triton streaming benchmark uses 640ms chunks and records 78–172ms average latency at 504–805 inferences/second; the same model in offline mode gives RTF 0.0009–0.0011 and 88–228 requests/second. Go non-streaming with attention rescoring for peak accuracy, shrink the chunk for peak responsiveness. Both are worth measuring on your own audio before you commit — boot a Tesla T4 ($0.298/GPU-hour) or an RTX 3090 ($0.193/GPU-hour) on NexGPU, connect over SSH or Jupyter, and you will have numbers within the hour. Bilingual support sits on Telegram with no ticket queue if you get stuck.

More in Speech recognition

Every model guide

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.

Sign up freeTalk to sales

Sign up to browse live network pricing. No payment method required.