Machine Learning and Music Classification: A Content-Based Filtering Approach
Using the Librosa Python Library, KNN, and Random Forest to Classify Music
In my previous blog post, Introduction to Music Recommendation and Machine Learning, I discussed the two methods for music recommender systems, Content-Based Filtering and Collaborative Filtering. The collaborative filtering approach involved recommending music based on user listening history, while the content-based approach used an analysis of the actual features of a piece of music. In this blog post, I will take a more in depth look at the content-based approach, using the Librosa Python library for “Music Information Retrieval” and trying a few machine learning classification algorithms to classify songs into genres based on their features.
Feature Extraction with Librosa
When I first started looking into the topic of music information retrieval, the process of extracting information about music and its audio content based on audio signal processing, it seemed like quite the daunting task that would require sufficient technical expertise. Well, it certainly does, there is even a whole field dedicated to the task. Luckily, that process has been made much easier by the creators of the Librosa Python library. Python users can use this library to easily extract information on any mp3 that you can feed it. Check out the video below that gives a brief tutorial by one of the library’s creators:
Librosa makes it easy to extract numerous features including beat tracking, mel scale, chromograms relating to pitch class information, and the ability to pull apart the harmonic and percussive components of the audio. Below I provide the code I used retrieve some of this information for the song Weird Fishes by Radiohead. Again, a good tutorial for all of these steps and much more can be found here.
First steps: Loading up the required modules
Mel-Power Spectogram
Chromagram
Pretty easy! Librosa really is a wonderful tool for music information retrieval. For the next step in my exploration of content-based filtering, I wanted to build an entire model that would be able to classify music to the correct genre based on a variety of features. For the data, instead of using the information provided by Librosa, which can take quite a bit of time and computational power if you are trying to analyze a large amount of songs, I decided to use information provided by Spotify’s Web API. Spotify provides song features that I can use for a classification model that are not as technical in nature as those provided by Librosa. For instance, some of the features include “Danceability” which “describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. ” and “Energy” which “represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy.” You can find a full list of the features, and their descriptions, included in my data in the next section of this post. The cool thing about this API is that it provides non-technical ways of describing a song that are derived from a more technical and scientific content-based analysis of the music.
Music Classification using K-Nearest Neighbors
Below I provide the code for my K-Nearest Neighbors classification model, where I attempted to classify songs into their correct genre. My data included about 300 songs, with about 1/3 being Hip-Hop, 1/3 Techno, and 1/3 Classical. I have also included the code on working with the Spotify Web API, which can be a bit tricky at first. Before you check it out though, here is a brief description of the features I received from the Spotify API:
Acousticness — A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
Danceability — Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
Energy — Energy is a measure from0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
Instrumentalness — Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
Key — The key the track is in. Integers map to pitches using standard Pitch Class Notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on.
Liveness — Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
Loudness — The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db.
Speechiness — Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.
Tempo — The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
Time Signature — An estimated overall time signature of a track. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure).
Valence — A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
A quick description of the process: I first requested the track id’s from three Spotify playlists, one playlist of each genre. I then requested the features for each track and combined all of the songs into one Pandas DataFrame. I then used the Scikit Learn K-Nearest-Neighbors model, looping through multiple K values to find the optimal one for my model. Ultimately that led to a K value of 9 which I used when fitting the model to the training data. Here are the scores for the classification model on the test data:
F1 score of around .93 for my test set. Looking at the confusion matrix, it seems like the model is having some trouble with the Hip-Hop songs, sometimes classifying them as Techno. Not bad scores, but let’s see if we can do better using another model.
Music Classification using Random Forest
Next I tried classification using a Random Forest model, an ensemble method that I hoped would get me more accurate results, using the same features I used in the K-Nearest Neighbors model. See code and results below:
Using Random Forest got me perfect classification scores! You can also see a bar chart displaying the importance of the individual features in the model. Clearly the Random Forest model was much more accurate than the K-Nearest Neighbors model, not surprising considering the simplicity of K-Nearest Neighbors .
So thats the end of my brief introduction to content-based filtering in music recommender systems. Genre classification is just one small part of this puzzle and I look forward to exploring other parts of these systems and the data of music.
