Effective Deep Learning Development Environment with PyCharm and Docker

Increase your productivity with a full-pledged IDE and virtualization.

Photo by Samuel Bourke on Unsplash

In this article, I will guide you through setting up an effective development environment for deep learning training when you use GPUs in a remote environment (on-premises workstations, AWS, GCP…).

At the end, you will be developing your ML/DL code in PyCharm IDE in your local computer as if it is using an Python interpreter in your local environment.

Motivation

1. Why should you use a full-fledged IDE like PyCharm?

source

“So if you want to go fast, if you want to get done quickly, if you want your code to be easy to write, make it easy to read.”
― Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship

1. You experiment a lot (literally, a lot) when you train your deep learning code. You change your hyperparameters, add/remove layers to optimize model, tweak your dataset… So, after a while, the code can be very messy to manage. Therefore, you need to write neat and maintainable code. An IDE like PyCharm can provide you this environment.
2. While coding, you have to observe your variables and functions in real-time to see if they behave as you expected, otherwise you may encounter unexpected results after long hours of training. It is very common to print the value of the variables or return value of functions, and most of the people do it by typing between lines. This is not a proper way to observe your variables; instead, you should observe them externally. It is effortless to see your code’s behavior and value of variables with PyCharm Variable Explorer and IPython Console.
3. It can be time-consuming to read the documentation of libraries to find proper functions or objects for our task. Auto-complete is very handy in writing fast code. However, it is tough to do it in a dynamically-typed language like Python. Although other IDEs and Jupyter Lab also have an auto-complete functionality, they are not always working correctly. PyCharm is very good at that job.

I think these three points are key factors for using PyCharm, but there are also many other great advantages: auto-formatting, advanced search, error checking, easy navigation, smart refactoring, profiling, coloring...

2. Why should you use Docker?

source

If you have set up a deep learning environment on an Ubuntu machine, you know how hard it is to match versions of all those drivers, software, and libraries. If you have not faced any difficulty, congrats! But you probably have not updated any of your drivers, CUDA, or deep learning (Tensorflow, PyTorch…) library yet, which means you can meet with this difficulty at any time.

Deep learning frameworks (especially TensorFlow) are very aggressive on version updating, which may lead to an incompatibility with the CUDA version or NVIDIA Driver. Sometimes, it is very hard to match the release of these softwares. In addition, if you accidentally install different versions of NVIDIA or CUDA drivers at the same time, it might be complicated to deal and you may need to format your system.

Docker solves these issues. You only install NVIDIA driver and the rest is managed by Docker containers.

For example, you are working on TensorFlow, but you have seen that in the new version of PyTorch, there is perfect functionality for your task, and you want to try it. It is challenging to manage different frameworks (and different versions, too) in the same machine. With Docker, you can manage it. It abstracts the system above the hardware driver; therefore, you can run different versions of the frameworks and CUDA at the same time. It is similar to a virtual machine but not as heavy as that. There are many advantages of using Docker, but in this article, I am not going to tell those. There are already many great articles on the web that you can read and get information extensively. This article is about connecting these two products. Let’s start.

Setting up the system

Flow diagram

1. Install PyCharm on your local computer (Professional edition is needed to use remote interpreter, PyCharm Professional is free for students).

2. Install Docker on your remote machine.

2.1. Get familiar with the Docker environment (it may take up to one work-day to fully understand what is image, container, how to fetch from Docker Hub, and how Docker system works in general, but you will be rewarded).

2.2. Download the Docker image of your favorite deep learning framework to the remote machine.

Tensorflow

PyTorch

Now, it is time to connect the PyCharm Python interpreter to the Python environment in the Docker container.

1. Start the Docker container. Forward port 8022 (you can change this) to 22 (it is a must for SSH connection). Example:

sudo docker run --gpus all -it --rm -p 8022:22 -v /home/workstation/Desktop:/home/gorkem/Desktop --name ssh_container pytorch/pytorch:1.4-cuda10.1-cudnn7-devel bash

2. Using the terminal of the main OS, check if you actually forwarded your port 8022 to 22 of container.

sudo docker port <your container name> 22
-> 0.0.0.0:8022

3. Set up SSH server in the Docker container.

apt update && apt install -y openssh-server
mkdir /var/run/sshd
echo 'root:<USE_YOUR_OWN_STRONG_PASSWORD>' | chpasswd
# Root password was changed with <USE_YOUR_OWN_STRONG_PASSWORD>
sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
echo "export VISIBLE=now" >> /etc/profile
service ssh restart

4. I was unable to SSH into root user in the container so I created another sudo user and established an SSH connection to it.

adduser gorkem
adduser gorkem sudo

5. Now, you should commit your container as a new image. Otherwise, you will lose all the updates you have made into your contanier when you terminate it. I generally add -ssh tag to the end of committed image.

docker commit <container_name> <name>/<name>:<tag>

6. Close the container.

7. Start a new Docker container from the commited image.

sudo docker run --gpus all -it --rm -p 8022:22 -v /home/workstation/Desktop:/home/gorkem/Desktop --name ssh_container pytorch/pytorch:1.4-cuda10.1-cudnn7-devel-ssh bash

8. Check SSH status, it will be disabled when the container starts (but you can set it to start automatically when the container starts), start SSH server.

service ssh status
->  * sshd is not running
service ssh start
->  * Starting OpenBSD Secure Shell server sshd
service ssh status
->  * sshd is running

9. SSH Server is running inside your Docker container, now you can connect your PyCharm remote interpreter to this container by following the official PyCharm guide. You should be careful on two issues:

9.1 Your username is the one that you created as user in Docker (in my example, it is gorkem).

9.2 Although SSH works on port 22 by default, you have to use the forwarded port (in my example, it is 8022). PyCharm is first connected to the remote machine, then the SSH connection is forwarded (by Docker) to our container.

Before directly connecting from PyCharm to Docker container, you can test SSH connection with a program like Putty (again, be careful about the port number and username) to be sure everything is OK on the container side.

Now you can develop your algorithm on PyCharm IDE as if your training is running on your local PC.

If you have any question or improvement, please comment.

Found this article useful? Check out my other articles below:

Deep Learning Architectures That You Can Use with a Very Few Data

How to Solve Data Loading Bottlenecks in Your Deep Learning Training