Music generation model

Riffusion self-hosted: about 4GB of VRAM in fp16, and one RTX 3090 keeps it real-time

riffusion-model-v1 fine-tunes Stable Diffusion 1.5 into a spectrogram painter: text produces a 512x512 image, and Griffin-Lim turns that image back into 5.12 seconds of 44.1kHz audio. Open weights, MIT code, and one real requirement — a card that finishes 50 diffusion steps in under five seconds.

Riffusion · self-hosted

Seth Forsgren and Hayk Martiros shipped Riffusion on 15 December 2022 with a premise that sounds like a joke until you hear it work: if Stable Diffusion can paint pictures, teach it to paint pictures of sound, then convert the picture back. riffusion-model-v1 is literally a SD 1.5 fine-tune — same UNet, same VAE, same frozen CLIP ViT-L/14 text encoder — so any inference stack that runs SD 1.5 already runs Riffusion. That is exactly why it remains the easiest on-ramp to self-hosted music generation.

You should know the current situation before you start. The riffusion.com domain now 301-redirects to producer.ai, which 301-redirects again to flowmusic.app. After raising $4M in October 2023 the team moved to a closed hosted product and has published nothing about the newer models — no weights, no parameter counts, no architecture. The GitHub repo riffusion/riffusion-hobby opens with a plain statement that it is no longer actively maintained, last touched 22 July 2024. The only Riffusion you can deploy yourself is the 2022 v1 line. That is not a dealbreaker, it just means using the thing for what it actually is.

And v1 is fully open. Code is MIT, weights are CreativeML OpenRAIL-M, and Hugging Face serves riffusion-model-v1.ckpt at 14.6GB with the full repo at 21.8GB. The official README names two specific cards — a 3090 or an A10G — as the tier that sustains the real-time experience. NexGPU stocks both: RTX 3090 24GB at $0.193/GPU-hour and A10 24GB at $0.414/GPU-hour, billed per second, compute billing stopping the moment the instance stops. Pulling the weights, running it end to end, and deciding whether it earns a place in your stack costs less than a coffee.

01 —

Which Riffusion versions exist, and which one you can actually run

One open-weights line; the hosted line has been renamed twice

Version	Parameters	VRAM	Context	Notes
riffusion/riffusion-model-v1 (weights)	~1B on the SD 1.5 skeleton (UNet 860M + VAE 83M + CLIP ViT-L/14 123M)	~4GB fp16 inference, ~8GB fp32; disk 14.6GB ckpt, 21.8GB full repo	512x512 image = 5.12s / 44.1kHz / mono	The only Riffusion with published weights. CreativeML OpenRAIL-M licence, repo last updated June 2023. No official fp16 or safetensors variant — just the raw ckpt plus the diffusers directory layout and a traced UNet.
riffusion/riffusion-hobby (inference code)	Python 3.9 / 3.10 only	Same as above; the code adds no VRAM of its own	CLI + Streamlit playground + Flask inference server	MIT licensed, and the README's first line says it is no longer actively maintained. requirements.txt pins diffusers==0.9.0, which is the first thing that breaks when you self-host — see the deployment steps below.
riffusion/riffusion-app-hobby (web front end)	TypeScript / Next.js	No GPU needed; can live on a CPU box	Endless stitching via the backend /run_inference	This is the infinite-scrolling music stream everyone remembers. Also MIT, also frozen. It chains 5.12-second clips using img2img looping plus prompt interpolation to keep the timbre continuous.
Current hosted line (Producer AI to Flow)	Undisclosed	Not self-hostable	Full songs with vocals	riffusion.com now redirects to producer.ai, and producer.ai to flowmusic.app. Closed SaaS with no published weights, parameter counts, or architecture. If you need this on your own hardware, you fall back to v1 or switch models entirely.
ACE-Step v1 (the open alternative we recommend)	3.5B	Peak reduced to 8GB after optimisation; bf16 plus --cpu_offload goes lower	Up to roughly 4-minute full tracks	Apache-2.0, and the most practical route to self-hosted songs you would actually publish. Official numbers: RTF 34.48x at 27 steps on an RTX 4090 (1.74s of compute per minute of audio), 12.76x on a 3090.
DiffRhythm-v1.2 (the other open route)	base / full tiers	8GB minimum; more if you disable chunked decoding	1m35s base, 4m45s full	Apache-2.0, and it takes lyric conditioning and reference-audio conditioning, producing a full structured song in one pass. Usually the first thing teams working in Chinese-language lyrics try.

02 —

Which GPU to rent for Riffusion

The official README names the 3090 and the A10G; we stock both

Get riffusion-hobby running and evaluate 5.12-second clips
RTX 3090 24GB$0.193/GPU-hour
The exact card the README names. It holds 50 steps at 512x512 under five seconds, 24GB is wildly more than a 4GB model needs, and it is the cheapest modern card on our price list.
Keep the Flask server resident, looping img2img to produce a continuous stream
A10 24GB$0.414/GPU-hour
The README's other named card. Passively cooled datacentre design, built for jobs that stay pinned and keep generating for hours.
DreamBooth or LoRA fine-tuning of spectrogram styles, or high-concurrency batch generation
RTX 4090 24GB$0.540/GPU-hour
Ample headroom for training the SD 1.5 backbone while demucs stem separation preprocesses in parallel, which cuts wall-clock time per training run noticeably.
Move on to ACE-Step 3.5B or DiffRhythm-full for complete songs
RTX 5090 32GB$0.723/GPU-hour
32GB lets you switch off chunked decoding and decode a full four-minute latent in one shot, which removes the seams at the chunk boundaries.

03 —

Self-hosting Riffusion in four steps

From bare instance to first audio; most of the clock goes to pulling 21.8GB

01
Spin up a 3090 on a PyTorch image
Pick RTX 3090 24GB in the NexGPU console and boot a prebuilt PyTorch image — the 2,000+ image library also carries ComfyUI and Whisper ASR, both useful later in an audio pipeline. First confirm the GPU is visible, then confirm ffmpeg is present: Riffusion needs it for every format that is not WAV.
```
nvidia-smi && ffmpeg -version | head -1 && python -c "import torch; print(torch.__version__, torch.cuda.get_device_name(0))"
```
02
Pull the weights, knowing what they cost on disk
riffusion-model-v1.ckpt is 14.6GB on its own, and the full repo including the diffusers directories and the traced UNet is 21.8GB. If you only plan to use the diffusers pipeline you can skip the .ckpt and fetch just the subfolders, which cuts the download roughly in half. Storage bills at a $0.414/GB-month median and keeps billing after the instance stops — until you destroy the volume.
```
huggingface-cli download riffusion/riffusion-model-v1 --local-dir ./riffusion-model-v1
```
03
Build a Python 3.10 env and route around the diffusers pin
riffusion-hobby pins diffusers==0.9.0, and 0.9.0 collides with essentially any post-2023 combination of transformers, accelerate, and torch. This is where self-hosting attempts most often fail. The safe move is an isolated Python 3.10 environment where those old pins can live undisturbed. If you do not need the Streamlit playground or demucs stem separation, the cleaner path is to load the weights as an ordinary SD pipeline on current diffusers — StableDiffusionPipeline.from_pretrained('riffusion/riffusion-model-v1', torch_dtype=torch.float16).to('cuda') — and write your own mel inversion.
```
conda create -n riffusion python=3.10 -y && conda activate riffusion && pip install -r requirements.txt
```
04
Start the inference server and make your first sound
The Flask server listens on 3013 and returns audio from a POST to /run_inference carrying a prompt and config. For something interactive, run the Streamlit playground on 8501, which ships ready-made pages for text-to-audio, audio-to-image, and prompt interpolation. Reach either through SSH port forwarding or the NexGPU web terminal — do not expose the port straight to the open internet.
```
python -m riffusion.server --host 0.0.0.0 --port 3013
```

How much audio per hour, and what it costs

Work it out using the README's own figure: roughly five seconds for 50 steps at 512x512 on a 3090, so 3600 seconds of wall clock yields about 720 spectrogram images. Each image is 5.12 seconds of audio, so 720 x 5.12s = 3686s, about 61 minutes of finished audio. RTX 3090 24GB is $0.193/GPU-hour, which works out to $0.193 / 61 = about $0.0032 per minute of audio — a third of a cent. Run an A10 24GB as a resident service instead, eight hours a day for a month, and that is 8 x 30 x $0.414 = $99.36. On storage, a 30GB volume for the weights at the $0.414/GB-month median is 30 x $0.414 = $12.42/month; that keeps billing after the instance stops, until the volume is destroyed, whereas compute stops the moment the instance does, metered per second. Egress runs at a $0.0081/GB median, so exporting 1GB of WAV is under a cent. No minimum, no setup fee, no quota request.

04 —

Frequently asked questions

How much VRAM does Riffusion actually need locally?

It is an SD 1.5 skeleton, so a single 512x512 fp16 inference peaks around 4GB, fp32 around 8GB, and a batch of four still lands in 6-8GB. Disk is the real appetite: 14.6GB for the ckpt alone, 21.8GB for the full repo. VRAM was never the bottleneck — throughput is. What you need is a card that clears 50 steps in under five seconds, and NexGPU's RTX 3090 24GB at $0.193/GPU-hour is the best value in that tier.

riffusion.com redirects somewhere else now. Is the model gone?

The model is fine; the company renamed. riffusion.com 301s to producer.ai, which 301s to flowmusic.app — after a $4M raise in October 2023 the team pivoted to a closed hosted product and stopped publishing weights. But riffusion/riffusion-model-v1 on Hugging Face and riffusion-hobby on GitHub are both still there under OpenRAIL-M and MIT respectively. Boot an instance on NexGPU and you can confirm it still works in about twenty minutes.

Why does Riffusion output sound muffled and metallic?

Three reasons, all sitting in the default parameters. max_frequency is 10000, so everything above 10kHz is discarded outright — that is the muffling. Phase is never modelled at all; it is reconstructed by 32 Griffin-Lim iterations, which is where the watery, metallic quality comes from. And stereo defaults to False, so output is mono. None of this is a bug — it is the inherent cost of the 2022 spectrogram approach. For clean production audio, go to ACE-Step or DiffRhythm; one NexGPU RTX 5090 32GB at $0.723/GPU-hour handles either.

Can it only generate 5 seconds at a time? How do I get a full track?

With default parameters step_size_ms is 10, so 512 pixels of width is exactly 5.12 seconds. The official approach loops img2img and interpolates in latent space between prompts so adjacent clips share timbre, then stitches them into a continuous stream — that is precisely how riffusion-app-hobby works. The seams never fully disappear. For a structured song in one pass, DiffRhythm-v1.2-full reaches 4m45s and ACE-Step reaches about 4 minutes. All three are one boot away on NexGPU, metered per second, so experimenting costs almost nothing.

Can I use Riffusion commercially? What is the licence?

Two layers. The riffusion-hobby code is MIT with essentially no restrictions. The weights are CreativeML OpenRAIL-M, a responsible-AI licence that permits commercial use but attaches a set of use restrictions, and the SD 1.5 base was trained on LAION-5B, so training-data compliance is a risk you must assess yourself. If you want that uncertainty gone entirely, Apache-2.0 ACE-Step and DiffRhythm are far cleaner. None of this is legal advice — read the terms yourself. On the compute side we impose no licence gate at all: 1,175 verified rentable nodes, take your pick.

Is self-hosting Riffusion still worth it today?

Yes, if you place it correctly. Its value now is pedagogical and textural — the spectrogram route is interpretable, visualisable, and lets you run img2img and inpainting directly on the image, a kind of hands-on control no end-to-end audio model offers, and it is cheap enough that cost barely enters the decision. For releasable songs, switch to ACE-Step 3.5B or DiffRhythm. On NexGPU you can run both tracks side by side: a 3090 at $0.193 for Riffusion experiments and a 5090 32GB at $0.723 for full-song generation, across 2,498 GPUs and 75 GPU models in 51 countries and regions, with bilingual support on Telegram and no ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.