HookTheory and the Music Theory Search Engine: MP3-to-Midi with Neo4J’s Graph Databases

There comes a moment in every person’s life when some great idea comes, followed by the sober realization that our technology is just not there yet.

Here’s the set of problems that I’m trying to solve for:

1. Musical innovation has (arguably) declined over the past decade. Apps like Instagram emphasize visual aesthetic over harmonic and melodic ideas, putting a financial incentive on musicians to shift toward refining their image and persona.
2. To become a better songwriter and composer, musicians need to be armed with knowledge.The emphasis on image over music produces artists that slide by on the bare minimum of music knowledge.
3. To become better listeners, an audience needs to be exposed to rich harmonic and melodic ideas, which is difficult in a climate of popular music where image dominates. The standard for musical innovation drops as the audience becomes musically illiterate and unaware of their own ignorance.
4. The solution seems to be an improvement to music education for songwriters and producers. The audience will meet artists wherever they are at and the standards will rise accordingly.
5. Music theory is taught primarily by academics who use classical music to discuss their ideas. Most songwriters today want to learn theory from modern music. There is a massive gap between academic music theory education and popular music today.
6. The emphasis on music production education corresponds to the rise in technology, while the inability of academic music theorists to catch up with popular music is largely responsible for the decline in literacy.

Q.E.D. To improve the quality of music in the world, a new method of music theory needs to be introduced that is both modern and accessible.

The Music Theory Search Engine

The solution that I propose is a music theory equivalent to Google.

1. A songwriter has a melodic idea and wants to think about ways in which it could be further developed.
2. The search engine, by means that are yet to be determined, would allow them to query all existing music to hear examples of how that idea has previously been developed.
3. In hearing the hundreds or thousands of options, the musician can pick the paths forward that suit their emotional and auditory aesthetics, while introducing new elements and building upon the existing collection.

A Path Toward Implementation

1. Audio-to-Midi that can transcribe full songs into multi-track midi.
2. A second interpretive layer of software would map over each of the midi layers and generate a queryable, many-to-many relational database based on the existing melodic, harmonic and structural devices.
3. All existing music would be transcribed by means of audio-to-midi and their queryable database content would be accessible to a search engine comparable to Google.
4. The search engine would allow users to submit a musical clip directly as audio or midi, map its devices, and cross reference with the existing storehouse to return a collection of results.
5. The data that returns would be presented as a graphic database using software like Neo4J,

Why This Isn’t Currently Possible

1. Audio-to-Midi technology (e.g. Melodyne) already exists, however it is generally ineffective for anything beyond homophonic melodies. In laymens terms, it can only handle one note at a time. This works for transcribing a vocal melody into midi, but fails in the attempt to transcribe full songs.
2. The concept of a musical device is similar to biology in its breadth. While geneology and taxonomy of species has been accomplished through a massive effort by the scientific community, it was largely done manually prior to the introduction of technology. A similar, centralized database of musical devices does not exist outside the gilded halls of classical academia.
3. There is only one existing system that I know of that queries midi data based on a taxonomy of musical ideas and provides a user friendly interface to promote research and education.

Taking the first steps forward

The most comprehensive existing solution to this problem that I’ve found is Hooktheory’s Theorytab web app. While they have not solved the problem of converting audio to midi or automating the transcription of midi to discreet musical ideas, they have created a community of people who take on this task manually.

Here you can see an example of a song as it appears in hook theory:

These songs are organized in a variety of ways:

Additionally, the site has a graph database that queries their existing chord progression data and allows the user to search based on a series of hand-selected chords:

HookTheory is a step in the right direction. If we could automate this process, the experience of studying music would be transformed forever and I suspect that the barrier to innovation (or at least harmonic and melodic complexity) would be dramatically reduced, leading to an improvement in the overall quality of music in the world.