Introduction
It is a common requirement in eCommerce, social media and other applications to need to match a subset of the total content with specific users as to maximize conversion, time in-app or other desired metrics.
A possible approach to determine what content to recommend to a given user is to recommend items that were well rated by peers that exhibit a tendency to rate content similarly. This approach is known as Collaborative Filtering. There are several different ways to calculate the similarity between user’s ratings of different items (videos, products or other content that is to be recommended). However, for this deployment we will be using Python’s SciKit Surprise which comes with built in algorithms.
Collaborative Filtering is often performed in 3 steps:
- Gathering of the data & pre-processing.
- Training of the model.
- Generating predictions.
In the next sections we will go over how to implement each of these steps.
Architecture and Setup
Pre-Requisites
Before we dive into the Lambda‘s code we need to setup the environment in which this Lambda will operate. A few assumptions on the setup of the application to which the recommendation algorithm is being added:
- The application is running inside a virtual private cluster (VPC).
- Data required for training the model is stored in a SQL database accessible from inside the VPC.
- The application servers have access (or can be given access) to S3.
Given the above assumptions, we can setup our Lambda so that it reads data from the SQL database and outputs results to S3. This ensures it runs asynchronously from the application servers which may be desired given the training and prediction are slow processes.
Lambda Configuration
We will be using [SAM](https://aws.amazon.com/serverless/sam/) to deploy the Lambda and its dependent resources. For a deeper introduction to SAM see the article below.
Let’s start with the folder structure. We will have a directory for the application source files, one for scripts that can be used for deploying the application, and top level files such as template.yaml that is used by SAM.
my-collaborative-filter/
├── cmd/
│ └── deploy.sh
├── src/
│ └── app.py
├── .gitignore
├── test-event.json
└── template.yamlThe template.yaml will look like the one below.
It defines 3 resource:
- The
RecommendationFunctionis the actual Lambda that will perform the collaborative filtering. - A
RecommendationFunctionSecurityGroupmay be required if your database has restricted access. The database security group could be configured to allow access from this specific security group. - The
OutputBucketwill be the S3 bucket we will output our results.
Take note at the Layers specification in the RecommendationFunction. Since Surprise and psycopg2 (PostgreSQL adapter) is not one of the included Python libraries with the Lambda OS we need to add it prior to being able to import it in the Lambda.
In the article below we go over a possible process for creating such a layer.
If you would prefer to build the layer without using Docker you can install Suprise in a Python virtual environment using an EC2 instance and copy the files over to the layer. The commands for installing the libraries on an EC2 instance are shown below:
python3 -m venv lambda_layer_packages
source lambda_layer_packages/bin/activate
pip install psycopg2 numpy scikit-surpriseLambda Code
The full Lambda code that would go in the app.py file is shown below with comments in-line.
Deployment
To deploy the Lambda simply run the deploy script below:
sam deploy
--template-file template.yaml
--stack-name recommender-stack
--parameter-overrides RdsPassword=YOUR_PASSWORD
--guidedLimitations
Since this implementation is designed to be simple it is important to discuss some of its limitations and how to improve on it:
Maximum Lambda Time Limit
As of the time of this writing, Lambda’s have a maximum execution time of 15 minutes. Running on a dataset consisting of 5000 items and 5000 users approaches the timeout limit.
Most of the time is spent in the predictions step since the code needs to execute for each user. Once the number of users reached is large enough such that the Lambda times out, we could instead have the main Lambda described above perform only steps 1 and 2 and save the output model (by pickling) to an S3 bucket. It would then enqueue (using SQS) a job for each user so that another series of Lambdas can run and make the predictions.
Maximum Disk Space
Lambda’s are limited to a maximum ephemeral disk space on the /tmp directory of 512MB. If the dataset as a CSV exceeds (or comes close to) this amount you may need to attach an additional volume to the Lambda or change the code so that the dataset is stored purely in memory.
Prediction for New Users
The above setup will only make predictions for users that have at least 1 rating. Depending on your setup you may have users with no ratings.
If you need recommendations for these users, you can recommend the globally best items. This can be done as a separate service entirely or (depending on the algorithm chosen for the collaborative filtering) you could make predictions for a user_id that does not exist (such as user 0) and default to that whenever there are no predictions for the user you are currently seeking.
The code below adds this functionality to the existing Lambda.
Non Real Time
This approach retrains the model at a fixed rate which needs to be at most as often as the Lambda takes to run. Therefore, it is not suitable for systems where being able to provide users with adaptive recommendations is necessary.
Conclusion
We have reviewed the deployment of a simple recommendation algorithm using Python’s Suprise library on AWS’s Lambda. The approach may be useful to test out the performance of the algorithm without devoting the resources to implement a full scale solution.
If you have any questions or just want to chat about startups, entrepreneurship, contracting or engineering just send me an email on paulo@avantsoft.com.br.
