Run Jupyter Notebook on the Cloud in 15 mins #Azure
The article does what the header says. Run a Jupyter Notebook on Azure Notebooks (Free or Paid) at a fraction of the cost.
In the pursuit to run the best data science models cheaper, faster and better, I sought out multiple avenues to compile complex code. I run models on Deep Learning, Neural Networks (RNN, CNN), Natural Language Processing (NLP) and Advanced Machine Learning (ML) often on my Github here. In my previous articles, I’ve compared multiple options for data scientists to run Notebooks on the cloud and am now deep-diving into one of my favourite ones — Azure Notebooks!
One of the biggest advantages of running Jupyter Notebook on the cloud is that you can have a decade-old PC/MAC that is incapable of locally running the latest deep-learning algorithm and still be able to deploy the most advanced machine learning models on the cloud in a jiffy, once you set it up with the help of this article. Read on!
The core issue with online cloud computing is that we are uncomfortable with the Cloud. The permutation of configurations, IPs, storage options, computation choices and connections overwhelm us.
I’m going to decipher the cloud for a Data Scientist.
The cloud is a difficult space to navigate, but, with a little help, you’re going to get exactly what you want — An extremely cheap and efficient method to run Jupyter Notebooks. I shall attempt to do the following:
- Set up Jupyter Notebook on Azure Notebooks
- Activate and navigate the Azure Portal
- Get Azure Data Science Virtual Machine (DSVM) set up
- Connect the DSVM to Azure Notebooks
This might be a long read, but, if your intentions are aligned with mine, you shall be satisfied at the end of it.
Get Premium Azure Services Complimentary:
Quick Side Note: Most people in pursuit of the same things are mostly working on a project from a reputed educational institute. So, I would suggest you avail Azure’s Student Benefits so that you can get 100$ of credits for free — Just use the email provided to you by your institute. I love the age we live in where cloud computing is cheaper than a can of coke, so head to Azure for Students and activate your Student account for free. It’s an easy process and won’t take more than a few minutes. Thank me later.
1. Set up Jupyter Notebook on Azure Notebooks
I have written a detailed step-by-step process for my fellow data scientists here that you can follow to set up and understand the nuances of setting up your Jupyter Notebook online.
The essence of it is mentioned below:
- Open Azure Notebooks, login and start a New Project
- Open a New Notebook and choose the kernel to run it on
- Choose the Free Tier option as compute
- Import required data and run on Jupyter Notebook
You should now have a Jupyter Notebook that can run on the cloud. The base configuration in the above-mentioned free tier is 1GB Storage and 4GB RAM. We shall work towards increasing the storage, cores, RAM and the computation power later in this blog.
2. Activate and navigate the Azure Portal
The account that you made the Azure Notebook with can be used to login to the Azure Portal here. You should now be able to see the Azure dashboard and portal. Let’s now have a quick walkthrough on the elements and terminology that we might be concerned with:
- Azure: Microsoft Azure is a cloud computing service created by Microsoft for building, testing, deploying, and managing applications and services through Microsoft-managed data centres.
- Virtual Machine (VM): A virtual machine is an emulation of a computer system. It’s essentially like having a computer in the cloud, but, instead of managing the hardware, you just rent the system as per your needs. Why?
This greatly reduces the cost of computation. As mentioned in my previous blog, this means that hardware that would have cost about 1500$ — 2000$ overall can be run in your decade-old laptop for a fraction of the cost. By fraction, I mean under 30 ¢/hour for a solid configuration.
- Cloud Services: Cloud computing is the on-demand availability of computer system resources, especially data storage and computing power, without direct active management by the user. Essentially, you only pay if the services are being used.
- Elasticity: The cloud is adaptable.
- Running simple data cleaning and feature engineering steps?
→ Great! You can use the free or lower-tier (cheaper) as computation.
- Running a complex model that needs more computation?
→ Great again! Just upgrade the system specifications, run the model for an hour and shift to the lower-tier again. - Blob Storage: Fancy terminology for Azure Storage. It’s a file system optimized to store structured as well as unstructured data that you can use for analysis in your Jupyter Notebook.
Remember, this blog does not teach you how to leverage all the functionalities of Azure, it is focused on aiding Data Science projects, especially Jupyter Notebooks to be deployed on the cloud via Azure Notebooks. If you are interested, I strongly recommend spending some time reading Microsoft’s great documentation.
3. Get Azure Data Science Virtual Machine (DSVM) set up
Navigate to the Azure Portal, you should see a screen similar to this:
We shall now create a Virtual Machine to connect to Azure Notebooks. The interface will be a Jupyter Notebook, where the computation will be present on Azure Services. Create a resource as shown below.
Search for Data Science Virtual Machine for Linux (Ubuntu) and select it.
You should now be looking at a screen similar to what is shown below.
You can configure the settings as per your need. I have filled out a basic sample that you can follow. Do not be daunted by the price as the prices are mentioned if you decide to run the system all through the month. We shall be running the VM for a few hours, not days, so, it will be an extremely small fraction of the cost that you can review before you proceed. If you are using Azure’s student subscription, then you get 100$ free for a year. Even if you are not, it will be extremely affordable.
Now select the Review + Create. On top, you should see how much the Data Science VM will cost per hour. For the above sample configuration (which is as good as some of the latest laptops in 2020 out there), the setup costs a mere 1¢/hour. Unbelievable! Based on your needs you can even perform deep learning modelling using the GPU for about 50¢/hour.
If you feel comfortable with all the mentioned specifications, just hit the Create button and we’re good to go. Else, go back and re-configure the specifications.
The page should navigate to a new page where it will take a few seconds or minutes (depending on the configuration to set up the VM)
Once deployed, you can see the following screen.
Here, I would recommend you set up auto-shutdown and download the deployment details on your local computer. I would also recommend keeping a back-up of the admin username and password for future reference.
Note: Running a Virtual Machine costs money. When you are not using it, stop it by selecting the Stop button. If the VM misbehaves, you can re-deploy it.
If you hit the Go to resource button, you can see the relevant options.
Good Job! You have now set up your Data Science VM with configurations that you would like to use.
4. Connect the DSVM to Azure Notebooks
Let’s summarize what you have done wonderfully so far:
- We have our Jupyter Notebook that runs on the cloud: Azure Notebooks set up, where we can already run a Jupyter Notebook on the free tier
- We have set up a Data Science Virtual Machine with the storage and computation that we require.
If you head to the Azure Portal, you should see the resource under Recent Resources as shown.
You have done a great job so far! We shall now proceed to do the step that we have been leading up to for so long: connecting our Azure Notebook with our Azure Compute.
Go to your Azure Notebook and select the option of running on Direct Compute as shown below.
You should now see an option to validate your Azure Virtual Machine Credentials as shown below.
For the IP, you can go back to the settings page of the Virtual Machine and copy the public IP as shown below.
A common error that I see is shown below.
Simply select the highlighted link and paste it in your browser. I believe this error occurs as we do not have a secure connection, but that’s alright as we are the only ones using it, so as much security would not be required for a personal deployment. However, if you are trying to deploy this for an enterprise, I would suggest you get in touch with your IT department to ensure that information security practices are followed.
Pasting the link should take you to a page where you are asked for your credentials. You can input the DSVM credentials and sign in.
Now, refresh the Azure Notebooks page and try the Direct Compute option again and hit Run!
You have now successfully deployed your Jupyter Notebook in the Cloud.
Conclusion:
As promised, you have successfully run Jupyter Notebook on the Cloud in 15 minutes. I spent a lot of time researching and thoroughly enjoyed writing this article. Show me some love if this helped you! 😄 I also write about the millennial lifestyle, consulting, chatbots and finance! If you have any questions or recommendations on this, please feel free to reach out to me on LinkedIn or follow me here, I’d love to hear your thoughts!
