Ruining Sudoku — A Data Science project (Part 4: Deployment and retro)
Photo by Hello I'm Nik 🎞 on Unsplash
If you stumbled by accident upon this article, I strongly suggest you to start from the first part of this series, for very obvious reasons.
This is it, the end of the journey as they (who?) say. The plan was to keep a kind of log of the activities required to go from an idea to a product, albeit a simple and limited in scope one, and so we did. And I respected the schedule!
Don’t thank me, true heroes don’t need that.
The other articles in this series
- Part 1: Introduction and Project Design
- Part 2: Data Preprocessing
- Part 3: Digits recognition and Sudoku solver
- Part 4: Deployment and retro — you’re reading this!
To quickly recap, at this point we have completed the data pipeline that goes from the image to the solution of the Sudoku puzzle. Today we close the circle by packaging our product by setting up a nice web-based frontend.
However, before we begin, a quick premise.
There are no such things as unicorns
And by unicorn I mean a person who is super good at all skills that are required for a data science project, that is project management, frontend and backend development, machine learning engineering and possibly more.
I consider myself quite skilled in Python programming and machine learning-related stuff, but for instance my knowledge about what’s the best way to deploy an application such as the one that I’ve been working on for this series is limited, to say the least. This means that sure, I can put something together that will probably work and not set the whole server room on fire, but it is very likely that any skilled <INSERT-JOB-TITLE-OF-PERSON-WHO-DOES-THESE-THINGS> would find my implementation suboptimal at best.
Therefore, don’t expect this specific part to be the best guide ever that you could possibly find on how to deploy a product, take it as a quick prototype that does the job in an overall decent way.
The [not so] dynamic duo: frontend + backend
Despite not wanting to spend too much time on this part (mainly because I’m not particularly efficient due to the aforementioned lack of expertise in this specific task), I still wanted to separate the frontend, that is the thingy that takes care of displaying stuff, from the backend, the part that does the heavy lifting (essentially all the code that I have described so far). Of course it’s a bit of an overkill here, but it fits the simulated scenario of “developing an app for real”. The final system will look like this:
I chose to use flask for the backend since I was already familiar with it and it plays well with the rest of the code written in Python, and Node.js for the frontend, since I never had an excuse to play around with it and this seemed a good one.
I really don’t want to go too much into details on how I did things since there’s really nothing particularly clever here, just your usual mix of HTML/JS/CSS (bootstrap) and asynchronous requests for the frontend, and the super standard inference function in the backend, plus a little bit of PIL magic to draw the solution (i.e., the missing numbers) over the original grid. This is how it looks like:
As usual, you will find the code in the repo for this project. Note that in order for the system to work you will have to either train a model yourself (following the instructions in the previous article of this series) or download the one that I trained (there’s a script called download_data.sh in the repo that does that). You will also need the .ttf font file used to write the digits on the output image (this is also taken care of by the script).
Dockerize ALL THE THINGS
If I were in charge, I would make it illegal to deploy things in any way that is not through a docker container or something similar. And I’m not talking about “you’re getting a fine” illegal, I’m talking life sentence at the very least.
Photo by Maxime Horlaville on Unsplash
So in order to make deployment as painless as possible, I created one image for each one of the two services (backend and frontend) and orchestrated them with docker-compose. Again, nothing fancy here, just the standard settings to expose the right ports and mount the folders that contain the required data (the model and the font).
That’s it, we’re done! One more thing, though.
Retro
It’s a good practice, once a project (or even a part of it) has been completed, to go over what went well and what went wrong so that one can improve and better next time. This is referred to as “having a retro” (short for “retrospective”).
What went well
I really feel like I deserve a pat on my back. Dedicating a good chunk of time to planning eventually paid out: once the list of things to do was decided, it was just a matter of organizing work using any task management solution that you feel comfortable with (I’m a huge Todoist hooligan) and of course finding the time to do that.
Everything went more or less smoothly with a minor exception that we will discuss in the next paragraph.
What went less-than-well
The only part that presented some unexpected challenges was the implementation of the digits recognition component. Without going too much into details (you can find a more in-depth discussion in the article dedicated to that part), the approach that I had originally planned to use didn’t work quite as expected, and it was necessary to find an alternative route.
To be honest, I saw that more like a “learning opportunity” rather than an actual mistake while planning, both because it is not really possible to foresee in advance issues with the data or with a given approach, and because it forced me to find an elegant solution that I wouldn’t have thought of otherwise, and this is an experience that I will carry over to future projects.
A final word from the author
I wrote this series to challenge myself to produce what I hope you will find to be quality content, with a decent amount of humor to make it tolerable. I tried (and apparently managed, yay me!) to keep a schedule of one new article every Monday, to avoid letting the momentum fade.
Please feel free to connect with me via LinkedIn, I’m always eager to discuss data science-related topics and such with anyone who shares this interest.
Moreover, if you are a data scientist cub and/or wannabe and have trouble figuring out what you should do to get a job in this fantastic industry, drop me a line: I’ll be more than happy to give you advice on this 🙃
Thanks for reading!
