The world’s leading publication for data science, AI, and ML professionals.

How to Connect to JupyterLab Remotely

How to work on JupyterLab running on a remote computer

Photo by Clay Banks on Unsplash
Photo by Clay Banks on Unsplash

Jupyterlab is a web-based interactive development environment (IDE) for Jupyter notebooks, code, and data. Connecting and running Jupyterlab from a laptop is straightforward. You simply type jupyter lab into your terminal and Jupyterlab will open in your browser, with the Notebook server running in your terminal.

What if you want to run it from a remote computer? How do you connect to Jupyterlab if it is running on a remote computer? This post will walk you through how to launch Jupyterlab on a SLURM cluster and then connect from a browser running on your local machine.

Table of contents


Installation

You can install JupyterLab with conda, mamba, pip, pipenv, or docker. The following command is used to install via conda.

conda install -c conda-forge jupyterlab

If you are not using conda I recommend the minimal and lightweight Miniconda installer. Here are instructions to get started using conda with Miniconda.

How To Use Miniconda With Python and JupyterLab


JupyterLab Security

To ensure other users can’t access your notebook, Jupyter Notebook servers can include a password for security,

To create a password, you first need to generate a config file:

jupyter server --generate-config

Running this command creates a configuration file at the following location:

~/.jupyter/jupyter_server_config.py

Then enter the following command to create a server password.

jupyter server password

After you enter a password a hashed version will be written to ~/.jupyter/jupyter_server_config.json . It is okay if the hashed version is public.

Now JupyterLab is secure and you will be able to log in with a password.


Launch JupyterLab on a remote computer

Now that JupyterLab is installed and secure, let’s launch it on a cluster’s compute node using the script below. I am using the SLURM scheduler, but this can be easily be adapted for other schedulers, such as PBS. Simply change the sbatch directives (#SBATCH) to PBS directives (#PBS).

It is good practice is to store scripts in your home directory on a high-performance computer (HPC). I prefer to keep them in ~/scripts/ . Since this is a sbatch script, we don’t need to make it executable like we would with a shell script.

You can submit the job to the cluster like this:

sbatch ~/scripts/launch_jupyter.sh

The SBATCH directives define the account to charge, the job name, how long to run the job for, and how the requested memory. Explaining SBATCH directives is beyond the scope of this tutorial. You can learn more about them here.

The line source activate base activates the conda environment I want to use. This can be changed to a different kernel within JupyterLab. How to make a conda environment into a kernel is discussed in this article.

The jupyter lab command has a couple of important flags attached to it:

  • --no-browser → Tells the server not to open a browser.
  • --ip "*" → Configures the notebook server to listen on all network interfaces and not just the default localhost/127.0.0.1
  • --notebook-dir /burg/glab/users/luke→ Tells JupyterLab to open in this directory.

Finally, once your job starts, server output will stream to SLURM-<JOBID>.out , where JOBID is a unique number for this job. The JOBID is displayed in the job status.

Job status

You can check the status of your launch_jupyter.sh job on a SLURM cluster with:

squeue -u <username>

Below is an example of output after running this command.

Here is what each column in the squeue output represents:

  • JOBID: This is the ID for the job [You need this if you want to cancel the job: scancel JOBID]

  • PARTITION: This is which user account is being charged [defined by #SBATCH --account=ACCOUNT]

  • NAME: This is the name of the job in the queue [defined by: #SBATCH -J NAME]

  • USER: This is the user who owns this job
  • ST: This is the status of the job. R means running
  • TIME: This is how long the long has been running for [Max run-time defined by: #SBATCH --time=01:00:00]

  • NODES: This is how many nodes you are using for this job [can be defined with a directive: #SBATCH -n NODES]

  • NODELIST: This is which node the job is running on. This will be useful in a moment.

Status codes

There a few status codes that squeue can list. When running JupyterLab there are only a few you need to be aware of:

  • PENDING (PD) → The job is waiting for resource allocation. It will eventually run.
  • RUNNING(R) → The job currently is allocated to a node and is running
  • FAILED (F) → The job terminated with a non-zero exit code and failed to execute.
  • SUSPENDED (S) → A running job has been stopped with its cores released to other jobs
  • STOPPED (ST) → A running job has been stopped with its cores retained.

KeyError: "runtime_dir"

This error was once an issue running JupyterLab on a cluster but is now resolved. However, if you run into this error, the way to fix it is to add the following line to your launch_jupyter.sh script:

export XDG_RUNTIME_DIR=""

Port forwarding Jupyterlab

First, we need to know which port Jupyterlab is running on. Most likely this will be port 8888. If you want to be sure, just look at slurm_<JOBID>.out . Below is the output of more slurm-77407.out , I highlighted the remote node and port in yellow. You could also grep for "http" to extract just that line

Now you can connect to this port like this:

ssh -L 8888:g026:8888 [email protected]

The first half specifies port forwarding and the second half specifies the user on the remote host.

ssh -L local_port:remote_node:remote_port user@remote_host

You have to keep this running in your terminal. You could run this in screen or tmux, just don’t forget about it.


Open JupyterLab in your browser

After inputting your password you will be able to open Jupyterlab from your local web browser. In this example, we opened it on local port 8888 so we type the following into our web browser:

https://localhost:8888

This will open Jupyterlab in your browser, but the commands will execute your remote HPC.

Password not working?

I highly recommend securing your server with a password, described in the JupyterLab security section. If your password does not work or you chose not to set up a password, then you can enter an access token instead. This token is shown in SLURM-<JOBID>.out. See the screenshot below.

You can access just the line with grep by finding just lines that contain "token".

grep token slurm-<JOBID>.out

Final thoughts

I love working in JupyterLab, especially if I am making figures and doing some exploratory analysis. I also enjoy working on an HPC, which provides larger storage than my laptop and access to compute nodes for big data processing. I hope this post helps you connect to JupyterLab remotely.

This cumbersome process will hopefully be obsolete as JupyterHub gains wider adoption. JupyterHub provides a way to serve Jupyter Notebooks to multiple users, which is especially useful on an HPC. However, it is not widely adopted yet. So until then, we will have to keep port-forwarding.


Please let me know if you have issues connecting to a remote computer, I am happy to help troubleshoot.

Thank you for reading and supporting Medium writers

Join Medium with my referral link – Luke Gloege, Ph.D.


Related Articles