Real-time music generation

Self-hosting Magenta: what a 2.4B real-time music model actually needs

Google Magenta's live line is Magenta RealTime 2. mrt2_base ships a 9.84GB fp32 checkpoint; mrt2_small is 1.13GB. Here is the honest VRAM maths, the real install commands, and the one limitation nobody mentions until you have already booted a card.

Magenta · self-hosted

Start with the thing that trips people up when they search for Magenta: the 19.8k-star magenta/magenta repo was archived read-only on 6 January 2026, with magenta-demos archived the same day. That repo is the TensorFlow 1 era — MelodyRNN, MusicVAE, GANSynth, Performance RNN — and today the dependency chain is harder to install than the models are to run. What is still moving inside the Magenta org is magenta-realtime, ddsp, mt3, magenta-js and magenta-studio, and for music generation the live line is magenta-realtime.

Magenta RealTime 2 landed on 4 June 2026 with two open-weight sizes: mrt2_base at 2.4B and mrt2_small at 230M. It is not a prompt-and-wait model. It is frame-wise autoregressive — one 40ms frame at a time, where v1 worked in 2-second chunks — which cuts control latency from roughly 3 seconds to roughly 200ms, about 15x lower than version one. The stack is SpectroStream (the audio codec) plus MusicCoCa (joint text/audio embeddings) feeding a decoder-only Transformer with a Depthformer. v2 also adds MIDI conditioning: a 128-dimensional multihot vector per frame, so you can play a keyboard straight into it.

One limitation has to be stated up front: true real-time streaming runs on the Apple Silicon C++ engine and the MLX backend. On NVIDIA you get offline (non-real-time) inference through the JAX backend of the Python library. So the right reason to rent a GPU is not to build a live synth in the cloud — it is bulk generation, MusicCoCa feature extraction, eval sweeps and downstream fine-tuning. Those are exactly the jobs a local Mac is worst at and a per-second-billed cloud card is best at.

01 —

What in the Magenta family you can actually run

Weights live at HuggingFace google/magenta-realtime-2, 15.6GB for the whole repo

Version	Parameters	VRAM	Context	Notes
Magenta RealTime 2 · mrt2_base	2.4B	9.84GB fp32 checkpoint; ~4.8GB of bf16 weights; plan on a 24GB card	25-frame windowed attention per layer × 20 layers, ~20s effective receptive field	The quality tier. On Apple Silicon it needs M2 Max / M3 Pro or better to stream live; on NVIDIA it is the right choice for offline batch generation.
Magenta RealTime 2 · mrt2_small	230M	1.13GB fp32 checkpoint; ~0.46GB of bf16 weights; 16GB is already generous	41-frame windowed attention per layer × 12 layers, same ~20s effective receptive field	The light tier. Streams in real time on any Apple Silicon Mac including the Air; in the cloud it is what you use for large prompt sweeps and ablations.
SpectroStream audio codec	Shipped inside the weights bundle	Part of the 15.6GB repo, resident on the same card as the LLM	48kHz stereo → 25Hz frame rate / 64 RVQ depth / 10-bit codes / 16kbps	Tokenises the waveform and reconstructs it. It is why output is 48kHz stereo rather than the 32kHz mono most open music models settle for.
MusicCoCa style embedding model	Shipped inside the weights bundle	Negligible when called on its own for embeddings	16kHz mono audio or text in → 768-dim vector, quantised to 12 RVQ tokens	Text prompts and audio prompts share one space, so style blending and timbre cloning both route through it. Callable without the LLM for bulk feature extraction.
Magenta RealTime v1 (legacy branch)	750M overall per the paper; LLM offered as Base 220M / Large 770M	HuggingFace google/magenta-realtime repo is 11.1GB (TF/Keras savedmodels)	10s of audio context, generated in 2s chunks	The paper reports RTF=1.8 on an H100 for the Large config, and demos running live on free-tier Colab TPU v2-8. Reach for it only to reproduce the 2025 results.
magenta/magenta, the classic Python library	MelodyRNN / MusicVAE / GANSynth / Performance RNN and friends	TF1-era models, CPU-runnable, VRAM is not the bottleneck	Symbolic MIDI rather than audio waveforms	Archived read-only on 2026-01-06 and kept only as a supplement to the papers. New work should start at magenta-realtime; use mt3 for transcription and ddsp for differentiable synthesis.

02 —

Which card to rent, by what you are actually doing

NexGPU list rates, metered per second and priced per hour; compute billing stops when the instance stops

Running mrt2_small, bulk MusicCoCa feature extraction, prompt sweeps
RTX 3090 24GB$0.193/GPU-hr
A 1.13GB checkpoint barely dents 24GB, and the 3090 is the cheapest card on the list that clears the sm_75 floor JAX CUDA 13 requires — cheaper than a Tesla T4.
mrt2_base 2.4B offline generation and bulk_generate runs
RTX 4090 24GB$0.540/GPU-hr
The 9.84GB fp32 checkpoint plus SpectroStream plus the JAX memory pool makes 24GB the sensible floor, and Ada throughput holds up far better than Ampere over hours of continuous generation.
Keeping codec, LLM and a fine-tuning copy resident at once
RTX A6000 48GB$0.817/GPU-hr
48GB fits mrt2_base, SpectroStream, MusicCoCa and optimiser state together, so you stop swapping checkpoints in and out and stop fighting JAX's preallocation.
Maximum single-card throughput, near-real-time offline pipelines
H100 SXM 80GB$3.582/GPU-hr
The only published GPU number in the Live Music Models paper is RTF=1.8 on an H100 (v1 Large config), so if you are capacity-planning against that benchmark, plan on the same silicon.

03 —

Getting Magenta RealTime 2 running on NexGPU

Python 3.12 + uv + jax[cuda13]. Four steps from boot to first audio.

01
Boot an instance and check compute capability first
Pick an NVIDIA instance at console.nexgpu.net — the Ubuntu CLI or PyTorch prebuilt image is fine, and no quota request is involved. First thing after boot, confirm the architecture: jax[cuda13] requires sm_75 (Turing) or newer, so Volta's Tesla V100 (sm_70) and Pascal's Tesla P40 (sm_61) fall outside the CUDA 13 wheels and need the legacy jax[cuda12] path, which supports sm_52 and up. CUDA 13 also wants driver >= 580.
```
nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv
```
02
Install magenta-rt with the matching JAX wheel
Python 3.12 is required and uv is the recommended environment manager. The base install ships CPU JAX only — you must add the accelerated wheel yourself, and skipping this is why people quietly run on CPU and conclude the model is slow. The [mlx] extra from the README quickstart is Apple Silicon only; do not copy it onto a Linux box.
```
uv venv --python 3.12 && source .venv/bin/activate && uv pip install "magenta-rt" "jax[cuda13]"
```
03
Pull the weights (note that this step is interactive)
mrt models init fetches shared resources, then mrt models download presents an interactive model picker — worth knowing if you were planning a fully unattended provisioning script; run it inside tmux or a Jupyter terminal. Weights land in ~/Documents/Magenta/magenta-rt-v2/, a genuinely surprising path on a Linux server, so match your persistent volume mount to it. Budget 15.6GB.
```
mrt models init && mrt models download
```
04
Generate audio, and rein in the JAX memory pool
JAX preallocates roughly 75% of VRAM by default, which blows up the moment you run multiple processes or share the card, so turn preallocation off first. mrt jax generate produces a 4-second sample to prove the path works; bulk_generate.py handles batch runs and writes to outputs/eval_audio/<size>/. For embeddings alone, from magenta_rt.musiccoca import MusicCoCa then m.tokenize(m.embed('a jazz piano trio')) returns the tokens.
```
export XLA_PYTHON_CLIENT_PREALLOCATE=false && mrt jax generate && python scripts/bulk_generate.py --size=mrt2_base
```

What one batch generation run actually costs

Take a mrt2_base batch run on an RTX 4090 24GB at $0.540/GPU-hr. Boot, image pull and mrt models init && mrt models download (15.6GB in total) take about 12 minutes, so 0.2 hr × $0.540 = $0.108. Then bulk_generate.py --size=mrt2_base runs for 3 hours: 3 × $0.540 = $1.62. Compute comes to 3.2 hours = $1.728, and it stops the moment the instance stops — no minimum, no setup fee. Output at 48kHz 16-bit stereo WAV is about 11.52MB per minute, so 100 minutes is roughly 1.15GB; at the median egress rate of $0.0081/GB that is $0.009 to pull it home, under a cent. Weights plus output occupy 40GB of storage: destroy the instance the same day and you pay a single day, 40 × $0.414 ÷ 30 ≈ $0.55, while keeping it a full month is 40 × $0.414 = $16.56 — storage keeps billing until the volume is destroyed. All in, a three-hour batch run with the download comes to under $1.80; if that run yields 100 minutes of audio, that is about $0.017 per minute of music. Running mrt2_small instead, an RTX 3090 24GB at $0.193/GPU-hr makes the same 3.2 hours cost $0.618.

04 —

FAQ

pip install magenta keeps failing — is the old Magenta repo dead?

Yes. magenta/magenta and magenta-demos were both archived read-only on 6 January 2026, and the README says outright that the repo now serves only as a supplement to some of the papers. It is TensorFlow 1 vintage, and the dependency chain is close to uninstallable on current Python. Go to magenta-realtime for music generation, mt3 for multitrack transcription, ddsp for differentiable synthesis, magenta-js for the browser. All three modern repos are ordinary Python packages that run out of the box on NexGPU's PyTorch prebuilt images, which saves you the two days you would otherwise spend wrestling TF1.

Can Magenta RealTime 2 run in real time on an NVIDIA GPU?

No, and the project is blunt about it: real-time streaming depends on the Apple Silicon C++ inference engine (magentart::core) and the MLX backend, and the hardware support matrix lists nothing but M-series chips. Both model sizes can run offline, non-real-time inference on any Apple Silicon Mac or NVIDIA GPU through the Python library's JAX backend. That is where cloud GPUs earn their keep — bulk generation, feature extraction, eval scoring, downstream fine-tuning, all the parts a local Mac is slowest at. NexGPU bills per second, so you stop paying the moment the run finishes.

How much VRAM does mrt2_base need? Is 24GB enough?

It is enough, and it is the sensible starting point. mrt2_base.safetensors is 9.84GB, which for 2.4B parameters is exactly fp32; in bf16 the weights come to roughly 4.8GB. What actually eats memory is everything around it: the SpectroStream codec, the MusicCoCa embedding model, and JAX's habit of preallocating about 75% of the card. Turn XLA_PYTHON_CLIENT_PREALLOCATE off and a 24GB card has room to spare. NexGPU's RTX 4090 24GB is $0.540/GPU-hr; if you want more headroom, the RTX A6000 48GB is $0.817/GPU-hr.

Can I run magenta-rt on the cheap Tesla V100 or P40?

It depends which JAX path you install. The current jax[cuda13] wheels require sm_75 (Turing) or newer, and the V100 is sm_70 while the P40 is sm_61 — both outside that range. The legacy jax[cuda12] wheels support sm_52 and up, so in principle they work, but you own the compatibility risk of an older stack. The simpler move is an RTX 3090 24GB: at $0.193/GPU-hr it is cheaper than the Tesla T4 16GB at $0.298 and architecturally far newer, and NexGPU has them available on demand.

Will the 4-bit / 8-bit quantisation help me fit mrt2_base on a smaller card?

Do not count on it. The quantised exports in the docs (mrt mlx export --bits=8, and --bits=4 --quantize-method=gptq --gptq-cal-steps=128) produce .mlxfn files for the Apple Silicon C++ engine, not for the CUDA path. On NVIDIA you load the raw safetensors through JAX, so budget from the 9.84GB fp32 figure. Rather than burning a day on a quantisation pipeline, rent a card that genuinely fits: an RTX 4090 24GB on NexGPU is $0.540 an hour, and one experiment costs less than the time you would spend.

Is the Magenta RealTime 2 licence usable commercially?

The code is Apache-2.0 and the model weights are Creative Commons Attribution 4.0 (CC-BY-4.0) — attribution required, commercial use permitted, which is unusually permissive for a music model. Training used roughly 71,000 hours of stock music, largely instrumental, and the model card explicitly prohibits generating content that infringes copyrighted works. To evaluate that licence on your own infrastructure and run a full compliance suite, NexGPU spans 51 countries and regions across 1,175 verified rentable nodes so you can pick a region that matches your data-residency requirements, with bilingual support over Telegram and no ticket queue.

Start building on NexGPU

Enterprise R&D team or solo developer — either way, your first job can be running in minutes.