Noise to Music
A song made of random notes. An optimizer nudges them, one step at a time, until they sound like the genre you pick. Press play and listen to it learn.
How it works
The model. A 16-bar song in A A B A′ form: a 4-bar section A, a contrasting 4-bar section B, and a final bar that closes the last A. Each bar has a melody, a counter-melody, a bass line and chords on a 16th-note grid. At step 0 every pitch, rhythm and chord is random.
The critic. A style model per genre, trained offline on about 2,800 real songs and jazz solos (roughly 300,000 bars). From each song it extracted the melody, the bass line and a chord for every half bar. It then learned how melodies move, which chord tones they land on at each beat strength, which rhythms fill a bar, how chords progress and how often they change, and how the bass relates to the harmony.
The loss. Each section is scored by how unlikely it is under the genre's model (negative log-likelihood), plus how far its statistics drift from the genre's, so it can't win by repeating one safe note. Each term is measured against held-out real music: a term reaches zero once a section is as plausible as the best 10% of real 4-bar phrases. A blend mixes the two models' probabilities.
Form and counterpoint. On top of that, the song is scored as a whole: the chords have to flow from A into B and back, B has to differ from A in tune and harmony, and the song has to end on the home chord. The counter-melody follows classic counterpoint rules: stay under the melody, avoid clashing with it on strong beats, avoid parallel fifths and octaves, and move when the melody holds still.
The optimizer. A genetic algorithm with a population of 200 songs. Each step mutates pitches, rhythms and chords, moves whole bars to a new chord, recombines bars from good parents and keeps what lowers the loss. It descends the loss like training does, only without gradients. Switch genre mid-run and it fine-tunes from where it is.
Add noise. Once a song has settled, Add noise scrambles a share of its notes and chords (light 15%, medium 35%, heavy 70%) in every member of the population, and the optimizer works the noise back into music. Light noise gives a variation on the same song. Heavy noise usually gives a new one with the same chords. Scrub the step slider back to compare. Start brand new noise throws everything away instead.
Longer songs. The 32-bar form adds a third section, C. Switching length keeps the sections already trained, and only C starts from noise.
Keep composing. While the song plays, new sections keep being written onto the end. No random notes are added here. Each new section starts from bars already in the song, with chords the genre model proposes, and is trained in the background to follow on from the previous section without repeating it. It joins the song only when it's finished, so you only hear finished sections.
What is learned. The notes and chords. Instruments, the comping rhythm and the backing drums are genre presets. Everything runs in your browser, and nothing is sent anywhere.
Training data: Lakh MIDI Dataset (Colin Raffel, CC BY 4.0) with tagtraum genre annotations (Hendrik Schreiber), and artist-labelled songs from its clean_midi subset. Jazz also learns from 432 transcribed solos, with their chord changes and bass lines, in the Weimar Jazz Database (Jazzomat Research Project), made available under the Open Database License. The trained models (models.json) contain information from it and are available under the ODbL too. There is no lo-fi MIDI dataset, so Lo-fi is learned from jazz and R&B under 95 BPM. The classical model is smaller (about 100 pieces) and noisier, because most of those files are human performances rather than quantized scores.
Next in the lab: evolving drum grooves, a skyline that grows into a city, and random characters that settle into a haiku. On iPhone, turn off silent mode to hear audio.