
Recommender systems have become ubiquitous in consumers’ daily lives on the online platform, ranging from e-commerce, social media to news outlets. Our preferences and biases are super-charged by machine learning algorithms that learn from our taste, and recommend more of what we desire to see. Because recommender systems have such profound impact on our lives, and consequently on society, I decided to spend my last project of the Metis Data Science Bootcamp (Singapore, Batch 5) on creating a simple recommender algorithm for an e-commerce web store. Before we take off on my project, I would like to give you an overview on two general types of recommender systems:
Content-Based Filtering Recommender: As its name suggest, this type of recommender uses the similarity among the background information of the items or users to propose recommendations to users. For instance, if User A generally gives Sci-Fi movies a good rating, the recommender system would recommend more movies of the Sci-Fi genre to User A. In another instance, if User B is an individual of a higher income group, a bank recommender system would likely label User B as a potential customer of the premium investment plan.
- Pros: Easy to overcome a cold start problem – when there is zero or few user-item interactions, the recommender is still able to provide good recommendations to the user.
- Cons: Requires the background information of items/users. Whenever there are new items/users, these background information has to be catalogued and added in.
Collaborative Filtering Recommender: This type of recommender identifies trends and patterns in previous and other user-item interactions, and advise similar recommendations to a present user based on his existing interactions. The core idea is based on clustering consumers of similar tastes together. For an instance, a news recommender detected that users who consume news favoring Donald Trump are also interested in news related to conspiracy theories. Hence, if there is a user newly interested in Trump news he will also be recommended news related to conspiracy theories.
- Pros: Exploits hidden correlations behind user-item interactions, and does not require extensive hand mapping and cataloging of data.
- Cons: Prone to cold start problem, and require existing reservoir of user-item interactions before being able to provide meaningful recommendations to existing user.
There are also hybrid Recommender Systems that incorporate both content-based filtering and collaborative filtering and achieved the best of both worlds. However, such recommender systems are typically advanced and require engineering a neural network.
For this project, due to a sparsity of data over a short time frame, I decided to employ content-based filtering for recommending products on an e-commerce web store, by customizing an algorithm that accounts for different categories of product information.
1. Data Transformation
I obtained about 110 million user-product interaction data of an e-commerce webstore over 2 months from Kaggle. Because of the enormity of the data, I decided to scale down the data to about 5 days, and filtering only ‘computers’ from the product category, which resulted in about 350k user-product interaction data left. The raw data set is shown below:
The column feature ‘event_type’ will serve as the target of my recommender model and has 3 categories: View, Cart and Purchase. I then proceed to provide a user score based on these user-item interactions – View: 1, Cart: 10, Purchase: 50. Furthermore, I also segregate items into 5 different price categories relative to their item categories. The assumptions in some of the models that are discussed later are that customers tend to shop in the lower-end or higher-end of products. The codes and resulting data frame are shown:
Note that each user-item entry are recorded for one user session, hence there could be multiple interactions per user per item. Therefore, I further perform a groupby operation to discover the sum of user scores for each unique user-item interaction. This forms the ‘ratings’ of the User-Item Matrix used for model building later. In addition, I apply MinMaxScaler to the user scores to obtain an interaction score with a value between 0 and 1. An interaction score of above 0.5 indicates a very high probability that a purchase has occurred, while no purchase occurs below the threshold of 0.5. The codes and resulting groupby data frame are shown:
2. Exploratory Data Analysis


Before model selection and development, I dive into data visualization of the target of the recommender system (View/Cart/Purchase). Unsurprisingly, the target is highly imbalanced, with about 1.8% view-to-purchase conversion. Moreover, a large majority of customers did not purchase anything at all, according to the histogram. This signals that it would be challenging for the recommender system to accurately predict purchase, as we shall see.
3. Model Selection
Proceeding to model selection, I first split the all ‘group’ data set into training, validation and test data sets using a Simple Validation paradigm:

Subsequently, I transformed the training set into a sparse User-Item Matrix, filling the empty entries as zero for user score:
Now, there are 3 different content-based filtering models that I will consider:
- Filtering by item category
- Filtering by item category and price category
- Filtering by item category, price category and brand
For each of the model, I will apply a matrix product between the User-Item Matrix and the Item-Item Similarity Matrix/Matrices according to the algorithm illustrated below:



Once the matrix multiplication is completed, the MinMaxScaler is once again applied to obtain the trained User-Item Matrix of predicted interaction scores. For simplicity of illustration, I will demonstrate the codes for Model 3, as shown below:
The trained User-Item Matrix is then re-converted back into a data frame, which is then merged with the validation data set. Again, items of predicted interaction scores above 0.5 are labelled as predicted purchase by customers.
Now, comparing the actual purchase and predicted purchase by customer for the three models, Model 3 appear to be the most promising with highest precision, without sacrificing too much on Recall as compared with Model 2. Nonetheless, the precision and recall for all models are very low, as expected from my analysis in the EDA process.

4. Model Evaluation
Choosing Model 3 as the final model for my recommender system, I proceed to re-train both the Training and Validation data set together, and then evaluate on the Test data set, achieving a Recall of 5.0% and Precision of 7.8%.


However, when I evaluate Precision and Recall based on number of unique purchases made, an interesting trend is noticeable. The Precision of the model is actually much higher for customers who have at least made one purchase! Hence, this means that when the model makes a recommendation to a buying customer, we are more confident that the customer will actually take up the product. The overall precision of the model is thus brought down by the false positive of predicted purchases of customers who eventually did not purchase.
Also, it is observed that Recall is much higher for customers who purchased more products. This could be because how model is measured is more biased towards purchasing customers. Hence, further tuning and adjusting could be done to raise the model’s Recall.
5. Model Demonstration
In this section, I will demonstrate the user-item interactions of a particular customer (User ID: 518044530) in the Training/Validation data set and Test data set and also the potential recommendations that the model will provide for the customer. The following user-item interactions are captured in the Training/Validation data set:
In the Test data set, we see that the model accurately predicted the user purchase. This is because in the Training/Validation data set, the customer had a high interaction score of a similar item.
Taking a random sample of the top recommendations of this customer (User ID: 518044530), we see that the model churned out products of similar category, price range and brand that he/she previously interacted with:
6. Conclusion
This project details the end-to-end pipeline of executing a Content-Based Recommender System, and is completed as part of the final project of Metis Data Science Bootcamp. Although the model is far from perfect, it showcases how a Content-Based Recommender can filter across more than one category of product information. In the future, ideally, a more effective hybrid recommender system could be attempted by building a deep neural network.
At the end of this 12-weeks bootcamp, I would really like to extend my heartfelt gratitude to my talented instructor Neo Han Wei, from whom I have learnt a great deal and also my fellow batch-mates especially Daniel Chang and Li Xinni who have lent moral and emotional support to make this learning journey possible.
Also, if you are interested, below are the links to some other projects that I have painstakingly crafted during the bootcamp:
Project 3
Predicting Satisfaction of Airline Passengers with Classification
Project 2
Predicting the Market Value of FIFA Soccer Players with Regression
Project 1
Exploratory Data Analysis – MTA Turnstile Traffic Analysis for Street Engagement
P.S. If you are interested to speed up your learning in data science, there is also an extremely useful article on building up good learning habits:
The Complete Guide to Effective Learning in Data Science
Lastly, thank you very much for reading! Here is the link to my GitHub, which contains all the codes and presentation slides for this project. Also reach me on my LinkedIn or comment here below to discuss!
Support me! – If you are not subscribed to Medium, and like my content, do consider supporting me by joining Medium via my referral link.