Magenta (project)

Magenta is an open-source research project from Google that explores the role of machine learning in creating art and music. Started by researchers and engineers on the Google Brain team and previewed in May 2016, it grew into a decade-long effort to build generative models, datasets, and creative tools that musicians, artists, and developers can use directly, with most of its code, models, and training data released publicly and, for most of its life, built on TensorFlow.^[1][2] Rather than aiming to replace human creativity, the project framed its goal as advancing the state of the art in machine intelligence for art and music generation and as building tools that extend an artist's workflow.^[1]

The project was previewed at the Moogfest music and technology festival in May 2016 and formally launched with a blog post on June 1, 2016. Douglas Eck, a research scientist at Google Brain, led the effort and authored the introductory post, which opened with the project's guiding question: "Can we use machine learning to create compelling art and music? If so, how? If not, why not?"^[1] From the outset the team committed to publishing models and tools as open source on GitHub, writing, "We'll use TensorFlow, and we'll release our models and tools in open source on our GitHub." That pattern defined Magenta for the next decade.^[1][3]

What is Magenta?

Magenta emerged from Google Brain at a moment when deep learning was rapidly improving image and speech generation, and the team wanted to test whether similar methods could produce compelling music and visual art. The project posed a deliberately open question: can machine learning create art that people find moving, and what role should the artist play alongside the model? Early work concentrated on symbolic music, training recurrent neural networks on large collections of MIDI to generate melodies and drum patterns. Over time the focus broadened to raw audio synthesis, controllable generation, and real-time interaction. The group operated partly as a research lab, publishing papers at venues such as ICLR and ICML, and partly as a maker of public-facing demos and instruments.^[1][2]

What has Magenta released?

Magenta released a steady stream of models, datasets, libraries, and instruments. Several were accompanied by peer-reviewed papers, while others were distributed as web demos or plugins. The most widely cited are listed below.

Scroll sideways for more →

NSynth, introduced in the April 2017 paper "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders," learns characteristics of individual sounds and synthesizes entirely new ones rather than simply blending samples. It built on the WaveNet audio model and was a collaboration involving Google Brain, Magenta, and DeepMind.^[5][6] The accompanying NSynth dataset comprises 305,979 one-shot instrumental notes sampled from 1,006 instruments, each rendered as a four-second 16 kHz clip across pitches and five velocities.^[6] The following year the NSynth Super, an experimental open-source hardware interface released by Google Creative Lab under the Apache 2.0 license, gave the algorithm a physical touchscreen instrument; it was never sold commercially but was used by artists including Grimes.^[7]

Music Transformer, detailed in a September 2018 paper (arXiv:1809.04281) and a December 13, 2018 blog post, introduced a relative-attention mechanism that, unlike the fixed-size hidden state of an LSTM such as Performance RNN, gives the model direct access to all earlier events. This let it generate coherent piano performances on the order of minutes and generalize beyond the length of its training examples.^[19]

DDSP, presented at ICLR 2020, took a different approach to audio. Instead of generating waveforms from scratch with a large autoregressive network, it made oscillators, filters, and reverberation differentiable so they could be embedded inside a neural network and trained end to end. This let the team produce high-fidelity audio with far fewer parameters and less training data, and it powered the Tone Transfer demo and, later, the DDSP-VST plugin.^{[11][12][13][14]}

What technology is Magenta built on?

For most of its history Magenta was built on TensorFlow, with Magenta.js using TensorFlow.js to run note-based models such as MusicVAE, MelodyRNN, and DrumsRNN directly in a web browser with WebGL acceleration.^[9] The symbolic models relied heavily on recurrent neural networks and variational autoencoders, while the audio work moved from autoregressive WaveNet-style decoders toward the lighter, signal-processing-informed designs of DDSP.^[5][11]

The project's more recent flagship models mark a shift in tooling. Magenta RealTime was trained using JAX and the T5X framework, with SeqIO handling data pipelines, on Google TPU hardware.^[15][16] This reflects a broader move within Google's music research, where the symbolic, TensorFlow-era tools of the late 2010s gave way to large audio models trained on JAX-based infrastructure, and, with Magenta RealTime 2, models designed to run directly on consumer laptops.^[20]

What is Magenta RealTime?

Magenta RealTime, sometimes abbreviated Magenta RT, was released on June 20, 2025 and described by the team as the first open-weights model capable of real-time music generation with real-time control. It is an 800 million parameter block-autoregressive transformer that produces a continuous stream of audio in short chunks, generating roughly ten seconds of context to synthesize each two-second block. A user steers the output live by supplying text prompts, audio examples, or weighted blends of the two, making it suitable for interactive performance and improvisation.^[15][16][17]

The system has three main components, building on the methodology of MusicLM. A codec called SpectroStream turns 48 kHz stereo audio into discrete tokens; a joint embedding model named MusicCoCa maps audio and text into a shared space to drive style control; and an encoder-decoder transformer generates the audio tokens.^[15] The model was trained on about 190,000 hours of mostly instrumental stock music. On a free-tier Colab TPU it generates two seconds of audio in roughly 1.25 seconds, a real-time factor of about 1.6.^[16][17] The codebase was released under the Apache 2.0 license and the weights under Creative Commons Attribution 4.0, distributed through GitHub, Google Cloud Storage, and Hugging Face.^[15]

Magenta RealTime is the open-weights counterpart to Lyria RealTime, a proprietary live-music model developed by Google DeepMind and offered through services such as Music FX DJ and Google AI Studio. The two were documented together in the paper "Live Music Models," posted in August 2025, which reports that Magenta RealTime outperforms other open-weights music generation models on automatic quality metrics while using fewer parameters.^[16][18]

What is Magenta RealTime 2?

Magenta RealTime 2 (MRT2) is an open-weights live music model released on June 4, 2026 that runs locally on Apple Silicon, removing the cloud TPU or GPU requirement of the first version. It ships in two sizes, a 2.4 billion parameter base model and a 230 million parameter small model, both released under the Apache 2.0 license on Hugging Face.^[20][21] MRT2 uses a decoder-only architecture with sliding-window attention operating on tokens from the SpectroStream codec, and replaces the first model's two-second chunks with frame-level autoregression, which the team says yields roughly 15 times lower control latency than version one.^[20] Real-time streaming of the base model targets M3 Pro or M2 Max Macs, while any Apple Silicon Mac can run the smaller model, and MRT2 adds MIDI alongside the text and audio control inputs of the original. The first model and its code were moved to a v1_legacy branch when MRT2 launched.^[20][21]

Why is Magenta significant?

Across nearly a decade, Magenta helped popularize the idea that generative models could be practical, hands-on creative instruments rather than purely academic curiosities. Its open releases lowered the barrier to experimenting with machine-learning music: developers could embed models in web pages with Magenta.js, producers could drop generative tools into Ableton Live, and hobbyists could build an NSynth Super or run a DDSP plugin at home. The research output, particularly NSynth and DDSP, influenced later work on neural audio synthesis and controllable generation.^[5][11]

The project also reflects the trajectory of music AI at Google. The original magenta/magenta repository was archived and set to read-only in January 2026, and active development shifted toward newer audio models and infrastructure, with Magenta RealTime and Magenta RealTime 2 serving as the open-weights bridge between the early symbolic tools and the large live-music systems of the mid-2020s.^[2][15][20]

References

1. Welcome to Magenta! - Google Magenta blog ↩
2. magenta/magenta on GitHub - project repository ↩
3. Google's 'Magenta' project will see if AIs can truly make art - Engadget ↩
4. Performance RNN: Generating Music with Expressive Timing and Dynamics - Google Magenta ↩
5. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders - Engel et al., arXiv:1704.01279 ↩
6. The NSynth Dataset - Google Magenta ↩
7. NSynth Super - Wikipedia ↩
8. MusicVAE: Creating a palette for musical scores with machine learning - Google Magenta ↩
9. Magenta.js - Google Magenta ↩
10. Magenta Studio: Free AI tools for Ableton Live - Ableton ↩
11. DDSP: Differentiable Digital Signal Processing - Engel et al., arXiv:2001.04643 ↩
12. DDSP: Differentiable Digital Signal Processing - Google Magenta ↩
13. Tone Transfer - Google Magenta ↩
14. DDSP-VST: Neural Audio Synthesis for All - Google Magenta ↩
15. google/magenta-realtime - Hugging Face model card ↩
16. Magenta RealTime: An Open-Weights Live Music Model - Google Magenta ↩
17. Google Researchers Release Magenta RealTime - MarkTechPost ↩
18. Live Music Models - Lyria Team, arXiv:2508.04651 ↩
19. Music Transformer: Generating Music with Long-Term Structure - Google Magenta, arXiv:1809.04281 ↩
20. Magenta RealTime 2: Open & Local Live Music Models - Google Magenta ↩
21. magenta/magenta-realtime on GitHub - Magenta RealTime 2 repository ↩