Image Recommendations with PyTorch + Flask + PostgreSQL + Heroku deployment
Packaging a PostgreSQL/ PyTorch based image recommender system with Flask, importing data and running it on the Heroku cloud application platform.
During the last weeks I have implemented an item-to-item image recommender system that uses PyTorch’s pretrained Resnet18 convolutional neural network to generate comparability through feature vectors, a database to manage the images and top-k lists and a user interface.
The developed solution and steps for the deployment on Heroku are presented in this article and the code is available on GitHub.
I have set and fulfilled the following requirements for this project
1. The web app is accessible on the internet.
2. The application has a gallery of images with pagination, loading only the relevant data for the page.
3. The data for the gallery and the recommendations is fetched from a database.
4. The recommendations are most similar images from a possibly large image set. They must appear instantaneously; this means that the similarity calculation happens offline.
5. The gallery is responsive.
6. The Flask application is suitably packaged for future extensions/reuse.
In my last post already parts of the requirements were realized as prototype, in particular 2 and 4. A Jupyter notebook is available at GitHub for the Resnet18 PyTorch recommender that takes an image as input and gives out most similar images from the same image set. To use the Flask app this notebook should be used to generate Pandas dataframes that are stored as pickle files. These are used as static content in the app.
In this post we will show how to realize also all other requirements, how to use an SQLite database locally and PostgreSQL on Heroku, how to use it for pagination, how to clean-up the work and to deploy it to Heroku to make it available on the internet via browser on PCs or smartphones.
You can access and test the demo app with a few images on Heroku through the following link:
This post is structured in the following sections:
1. The features of the app are described
2. The calculation of similarities between images is revisited
3. The packaging structure for the application is described
4. The SQL model is presented
5. The backend application logic is introduced
6. The templates are introduced
7. All steps for the Heroku deployment are explained
8. A conclusion is made
Note, for this article I was using a Windows 10 machine and Python 3.5.4. I have left out any exception handling.
1. Features of the app
To get a better understanding what the implementation is for, let me briefly explain the features of the application.
Image gallery:
One can browse a set of images in a self-written gallery, showing all the images from the initial input set (a small test set generated from my private image repository), as depicted below:
The gallery has a custom design that was created with stylesheets. It defines the sizing of the header, the mouse reactions, the images, the text overlay, and pagination number appearance.
Pagination:
You can see the numbers below the images in the gallery. These allow to select a page with images. To not load all data at once I moved away from a JavaScript based approach that lead to hiding more information, but it was still loading all images. With the presented implementation, only the relevant data is loaded.
Image recommendations:
To each image from a possibly large image set, similar images with similarity values are fetched and presented in a second route. A one-to-many relationship is used for that end to show the most similar images to a selected input. The approach allows to change the number of sought recommendations. We use four, due to the small, artificial image set.
As the images are truncated you can further click and enlarge them in a modal to see the complete images, this modal uses some basic JavaScript.
Responsiveness:
The gallery and the recommendations are responsive, this means that you can use it on a smartphone and the image selection and proposals are user friendly on such devices.
Database commands:
I prepared some test data and used a dictionary to use it in an earlier prototype. This is reused for an import command. One can use the console to create or clear the database, and to import data described in such a dictionary. This works locally for the SQLite, but also for the PostgreSQL database at Heroku. The commands are included as Blueprint module.
2. Calculating recommendations offline
In a previous article I explained in detail how one can easily generate a similarity matrix and top-k lists (for each image, the k most similar other images). An implementation was presented in a separate post. Hence check-out these posts for details.
What we do, briefly:
- Preprocess images from an image input folder
- To each image create feature vectors with Resnet18 from PyTorch
- Compare feature vectors and store top-k lists
This is done in one Jupyter notebook, where I now added two lines to store the dataframes with similar image names and similarity values as pickle files. These are used as static content for the recommender application presented here. You can check-out the repository with the test images and it should run without efforts.
What we need to add is the import of these results — the actual recommendations — to an SQL database.
For the online processing the backend process only queries the most similar images for rendering the proper html.
3. Packaging the app
The following application structure has been used:
~/imageRecommender
|-- requirements.txt # All packages to run the app
|-- Procfile # file needed to use gunicorn
|-- run.py # starts the application
|-- .flaskenv # sets environment variables
|__ env # virtual environment – not in vc
|__ /imageRecommender # The application package
|-- __init__.py # Application factory method
|-- config.py # Configuration file for dev/prod
|-- models.py # Database models
|__ /commands # For import to database
|-- __init__.py # Application factory
|-- commands.py # Commands for DB creation/import
|__ /main # For routes
|-- __init__.py # Application factory
|-- routes.py # Business logic, fctns for routes
|__ /static # All static data
|__ /css # Stlyesheets
|__ /javascript # Java Script code for modal
|__ /pickles # The top-k lists in pickle format
|__ /site_imgs # The images for the app
|-- favicon.ico # The favicon
|__ /templates # All templates
|-- home.html # Home route
|-- layout.html # General layout
|-- recommend.html # Recommend routeThe __init__.py files make the folders that they are contained into modules. These modules can be registered as Blueprints. Only the __init__.py file on the imageRecommender level is non-empty in this application.
4. The data model
We store the image information in an SQL database. Therefore, we use Flask-SQLAlchemy, a library that helps using Flask with SQLAlchemy — which again is a Flask SQL toolkit that gives you the possibility to use SQL with Flask.
3.1. Getting started with SQL for Flask
The package supporting SQL databases is flask-sqlalchemy, version 2.4.4 is in the requiremets.txt file. With this we can import it to the application in the factory for the imageRecommender application factory __init__.py.
We configure the SQLite database for the Flask application locally and a Postgres database (everything is stored in one file then) for Heroku in the config.py file
Dependent on the use-case (development or production on Heroku) the right configuration is chosen. The secret key in production needs to be unique for security and the URL for the database needs to be set.
3.2. The data model
We create a class for the images for the gallery and a related class for the images to be recommended.
Some remarks to this model:
- The id identifiers for both tables are set automatically and uniquely as default
- imageName and imageDescription are strings for the metadata of the images
- imageRecs is the relationship to the recommendations, these are needed for the recommend route
- The Imagerecommendations class contains the related ID from the queried image, the name of the recommended image and its similarity value. It denormalizes the normal form with respect to the image name
We can now create the database and import data.
3.3 Database commands
There are three simple commands:
- to drop the database
- to create it
- to import data
The app is packaged as described above such that the commands.py script has access to the application — we will see that when talking about Blueprints later.
After running the import one has already a database that one can test, e.g. by creating queries.
5. The backend application logic
It was already described in my last article what Flask is and how to set-it up.
The solution is packaged to the imageRecommender package, and we call it with the run.py script
The createApp method is in the __init__.py application factory, and it becomes useful to work with it when extending the application, e.g. by tests that require several instances of the app.
- For the imports the configurations are loaded, these were described earlier. The app needs to be created in this __init__.py file via app = Flask(__name__).
- After the object is created the environment variable is validated for the right use-case and the suitable configuration is loaded.
- The database object is created, and two Blueprints are registered for the main application and the database commands. cli_group=None merges the commands to application level, so that they are just callable by flask <command name> in the Heroku console.
The main Blueprint contains the main backend application logic in the routes.py file. Let’s go through this file set-by step:
from flask import render_template, request, Blueprint
from imageRecommender.models import Galleryimages- We work with templates that are used for html page generations through the render_template package.
- Via request we get the name of the selected image and the page that we are on in the gallery.
- As we use Blueprints for modularizing the application, Blueprint is imported.
- To the queried image we get the similar images via the Imagerecommendations related class, and therefore we work with our Galleryimages class.
main = Blueprint('Fmain', __name__)- The blueprint Fmain is registered.
@main.route("/")
@main.route("/home")
def home():
page = request.args.get('page', 1, type=int)
gImages = Galleryimages.query.paginate(page=page, per_page=8)
return render_template('home.html', images = gImages)- The main and home route give access to the gallery.
- Via the home.html template we render the images given within the render_template call.
- It takes as input the images from gImages. Here the pagination feature is used. With per_page it can be set how many images are to be shown on one page of the gallery, and page is the default page to be opened, in this case the first page of the gallery with the first eight images.
@main.route("/recommend")
def recommend():
selectedImage = request.args.get('selectedImage')
imageEntry = Galleryimages.query.filter_by(imageName=selectedImage)
images = []
values = []
for image in imageEntry:
for recommendation in image.imageRecs:
images.append(recommendation.recommendedName)
values.append(recommendation.similarityValue) return render_template('recommend.html', title='Recommendations', customstyle='recommend.css', inputImage=selectedImage, similarImages=images, similarityValues = values)if __name__ == '__main__':
app.run(debug=True)
- The recommend route is called with the corresponding recommend function that requests the user’s selectedImage.
- It is queried from our database, and we iterate all recommendations from the related class.
- The image and value arrays are filled with corresponding information.
- All required information is passed to render the recommend.html page.
6. The templates
We saw two calls to render html pages from templates:
render_template('home.html', images = gImages)render_template('recommend.html', title='Recommendations', customstyle='recommend.css', inputImage=selectedImage, similarImages=images, similarityValues = values)
Let us have a look on the relevant sections of the home.html and recommend.html files
The application uses a layout template layout.html to maximize reuse of html code. The important part is that the body of the template has a block content that is specified within the other two html files.
<body>
…
{% block content %} {% endblock %}
</body>The home.html file is used for the gallery
It consists mainly of two parts:
1. The images in the gallery that are iterated via {% for image in images.items %} and
2. The pagination elements in the pagination div section. Here the most interesting part is the iter_pages method that allows to set how many numbers are shown to the left and right edges, and in the middle of the number block
{% for page_num in images.iter_pages(left_edge=2, right_edge=2, left_current=1, right_current=2) %}
The recommend.html template is used to show the query and the most similar images:
The modal at the end and the inclusion of the only JavaScript code in this app are for the pop-up that enlarge the similar images if wanted.
Responsiveness:
To properly show the gallery on a smartphone some design elements need different sizing and the gallery shall have only one column. This can all be realized with stylesheets by adding a corresponding section
@media only screen and (max-device-width : 640px) {and for the gallery content we have set for example the column-count to one in this section
.gallery .galleryContent .item{
float: left;
width: 100%;
height: 100%;
position: relative;
cursor: pointer;
column-count: 1;
}7. Heroku deployment
The .flaskenv file contains two entries and it works when python-dotenv is installed
FLASK_APP=run.py
FLASK_ENV=developmentso that flask run lets you test the application locally on http://localhost:5000/
Now it is time to deploy the package and make it accessible through the internet.
Heroku is a platform that allows to easily deploy and host applications. It spares most efforts, no usual system administration, webserver, firewall set-up tasks etc. are needed. A free tier allows simple tests and the presented demo runs on such a free host. We want to deploy the application to Heroku as it offers free cloud web hosting services that are optimal for a demo as this one.
The following steps are tested and working in August 2020 on Heroku.com.
For a simple deployment to Heroku push your code to GitHub. Create a repository. Use the URL you received to push the repository to GitHub.
Therefore, you need to visit https://www.heroku.com/ and create an account with the primary development language Python.
To install the Heroku CLI follow the instructions on https://devcenter.heroku.com/articles/heroku-cli.
You can carry out many of the following steps with the Heroku CLI, however, if possible, I present UI based configuration steps.
In Heroku click on “Create a new app”. It requests you to give it a name, let us set here image-recommender-demo and the region Europe.
Confirm and you get to a next screen, where you can select your deployment method
As we pushed the code to GitHub, we select the GitHub option, after which we need to “Connect to GitHub” and authorize Heroku to access your GitHub repository.
Next, we enter the name of the corresponding repository image-recommender-demo, search for it and connect to it.
You can select to automatic deploy each change, e.g. to your master branch, which is very handy for such demo use-cases,
For the start we deploy it once manually (there is another button to be pushed for that end). You can now see your solution being built and the state of success
On top of the site you can find a button
That leads us in this case to our recommender system on
https://image-recommender-demo.herokuapp.com/
Until now the logs will show that no web processes are running, therefore a Procfile needs to be created, this is used by Heroku to understand how to run the application. It contains only the line
web: gunicorn run:appweb specifies that a web worker is used. An application server is needed, therefore gunicorn is used and it needs to be installed — version 20.0.4 is referenced in the requirements.txt file. gunicorn is the command needed to run the web process, such that the python code can communicate to the webserver. run refers to the run.py file containing the application object, app refers to the name of the application object.
After this is committed, one needs to set FLASK_ENV. As mentioned I use the .flaskenv file locally. In production the configuration is set manually via the Heroku CLI
heroku config:set FLASK_ENV=production --app image-recommender-demoIf you are not yet logged in, Heroku will request a login first.
Now running the app still leaves you with an internal server error as the database is not set-up yet.
We want to work with PostgreSQL. Click on “overview” in the Heroku UI, select “Configure Add-ons”, search for “Postgres”; select “Heroku Postgres”. I use the free version for this demo.
Now this database is already attached to the app image-recommender-demo! You can see it now under “Settings”, “Config Vars” with its URL.
If psycopg2 was not yet installed Heroku will tell us again an error and the logs reveal that the module “psycopg2” was not found. Therefore, psycopg2==2.8.5 in the requirements.txt file gets the right package.
The tables still do not exist, so we will still see an error after this adjustment on Heroku.
To create the missing tables and import some data we use the commands from the command.py file explained earlier.
We shall set the FLASK_APP variable such that the commands are recognized in the Heroku con
heroku config:set FLASK_APP=run.py --app image-recommender-demoNow we can call the console in Heroku and use the commands presented earlier
flask createDB
flask importDBThat’s it, the application is deployed, online and the database contains relevant data!
8. Conclusions
With this and the PyTorch recommender post you have everything at hand to
- Run machine learning code offline and store the results
- Create a Flask based web application
- Use the machine learning results in the web application
- Use a PostgreSQL database with Flask
- Deploy your application to Heroku
You are not bound to use these programming insights for this particular demo app, you can become creative with other ideas once you understood all the steps.
There are still further aspects to be considered as described in my last article to aim for production.
We could invest some more work and make the recommendations itself an API service with suitable endpoints for adding, deleting, and updating the entries.
This kind of service could be a part of a production system like a larger online shop. Note that in this case the images should be moved from the static folder to for example some object storage.
Thanks for reading! Liked the topic?
If you found the read interesting, you might want to catch up on my previous articles on the topic that this article extends upon:
