The world’s leading publication for data science, AI, and ML professionals.

Run Jupyter Notebook as a background service on Ubuntu

Never lose a remote notebook server again, and even let the notebook server be started with the system on reboot

Photo by Christopher Gower on Unsplash
Photo by Christopher Gower on Unsplash

How to run Jupyter Notebooks in the background, not needing to keep a terminal open for them? In this article, I will show you two ways, a simple one-liner with nohup and a more advanced one using a daemon service.

Avoiding the need to keep a terminal open for your notebooks can be useful when running locally, but even more so on remote notebook servers where temporary disconnections are possible. What if we could just run the notebook server as a long-lived background task as a system service? So that we can connect to it any time and it even restarts on any failure or reboot of the machine.

This tutorial assumes that you are on a unix based system, I am on ubuntu and have tested versions 18.04 and 20.04. The second way requires root permissions, the first does not.

The simple one-liner

Instead of running jupyter-notebook ... with any options like for specifying the port used, you will want to use the following, in the directory where you want your notebook’s root to be:

nohup jupyter-notebook --no-browser --port=8888 &

Note the & character at the end of the command, which makes it run in the background. Any output will be written into a file called nohup.out by default and you can inspect the notebook logs there.

Like any other process, this has a PID and can be later stopped using that. (Normally jupyter-notebook stop <port> is not working with this method.) The PID is printed at the execution of the above command, though on-screen only, but there is no need to make a note of it, because you can find it later:

You can either check the PID of processes that have your output file opened:

$ lsof nohup.out
COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
jupyter-n 44361   username    1w   REG   0,51      884 19846171 nohup.out

or use ps to find the process between all the running processes of the computer:

$ ps au | grep jupyter

finally, use

$ kill -9 <PID>

to kill the process.

This is relatively simple and allows you to check the logs at any time in that directory, as well as keep the notebook running even if a terminal session ends. There is no need to have root permissions, though the server does not restart on a reboot of the machine either.

Photo by Avi Richards on Unsplash
Photo by Avi Richards on Unsplash

Daemon process on the system

If you have root permissions, you can make the notebook server run as a system service and be even started on reboot. This is for linux-based systems, using the built-in service manager.

Step 1: find the path to your jupyter-notebook executable

You need to do this having activated the correct python environment.

$ which jupyter-notebook
/home/username/.local/share/virtualenvs/notebook/bin/jupyter-notebook

(optional) Step 1.1: Set notebook password and any other config

You will find it easier to use the service down the line if you rely on a secure password set rather than copying the tokens that the server generates on startup.

$ jupyter-notebook password
[... password prompt, type it in twice] 
[NotebookPasswordApp] Wrote hashed password to /home/username/.jupyter/jupyter_notebook_config.json

You will need to type the password twice, which is then stored in a seeded format into the local jupyter config directory.

Step 2: Write the service settings file

The system services are specified in INI-like files for their settings and properties. These are stored in the /etc/systemd/system/ directory.

Let’s write a service file! This is a relatively simple one, where we have specified the executable for jupyter and the parameters, including the directory to start the directory for the notebook which is the same as where the service will be executed from. Change the user and group settings to your own user, make sure it is not root.

Notice the Restart section, this defines that the service should be restarted if failure occurs and the daemon waits 60 seconds with this.

Step 3: enable and start the service

We will want to place this service setting file into the correct system directory, then "enable" and "start" the service. Enable means that the service is started when the computer restarts, otherwise you need to manually start it.

# add the service file
$ sudo cp jupyter-notebook.service /etc/systemd/system/
# let the daemon reload all the service files
$ sudo Systemctl daemon-reload
# (optional) "enable" -> started when the computer boots
$ sudo systemctl enable jupyter-notebook.service
# start the service
$ sudo systemctl start jupyter-notebook.service

Once this is done, you can look at the service status. Note that this does not require root permission to display the status.

$ systemctl status jupyter-notebook

In case you want to stop the service, you can use sudo systemctl stop ... and for disabling start on boot use sudo systemctl disable ...

Step 4: Enjoy!

Now you have a notebook daemon service that runs on reboot as well and you can access it any time. The system takes care of restarting it in case the service exits and you can get back to work to never worry about it.

For example, if you run your notebooks on a remoter machine, like your home/work computer or in the cloud, you only need to connect to it through SSH and have the notebook up and running.

Conclusion

We have seen two methods for running a Jupyter Notebook server in the background, a simple one for non-root users and one more advanced one for root users. The first needs a little chaperoning for starting and stopping, while the latter should have you covered on a server once for all.

I am using both methods on various computers, depending on the user privileges I have. For example, the first is best on scientific computing clusters, while the latter is amazing for an AWS EC2 instance that is kept for notebooks and data analysis to be done from anywhere.

Photo by George Kroeker on Unsplash
Photo by George Kroeker on Unsplash

References & useful links

  • man systemd.service and man systemd.unit provide comprehensive documentation of the system service file options, and this is a digest thereof
  • kudos to this tutorial which I used long ago to set up a system service for the first time, and adapted later on multiple machines

Related Articles