Kubernetes Practical Intro for Data Scientists

Yup, it’s yet another “intro to k8s” article. The world didn’t need it, you didn’t ask for it, and yet I already had it in a gist somewhere so I thought: “why not”?

I presented this tutorial live recently for New York City Data Science Academy, it was recorded here. I’ve adapted the slides and gist to this single post for convenience. For folks interested, I’ll also be teaching a Machine Learning in Production class in 2022–23!

Me codin’ up k8s stuff with the boys — Image by Author

This post has two portions, a high level overview and a hands on portion where you deploy an ML model to k8s right on your laptop. If you just want the code for getting started locally, see the gist or skim to the second section of this post!

Overview

This first section outlines some high level Kubernetes knowledge that will give color to the practical portion.

Learning Objectives

By the end of this post, you should be able to:

• Understand why people use Kubernetes
• Understand why people wouldn’t use Kubernetes
• Describe some basic buildings blocks at the high level
• Deploy a simple ML app on a local Kubernetes cluster

We will NOT be covering:

• Kubernetes internal architecture
• Cluster setup and management
• Or really anything super in depth and out of my league

What is Kubernetes?

“Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.” — The K8s Website

Ok? But what does that even mean? To explain why we might want to use Kubernetes, let’s look at a simple example of deploying and managing an application without it.

Deploying and Managing an Application Without Kubernetes

Here’s a dramatically oversimplified workflow:

1. You build a simple app to search news articles
2. You deploy the search app to a single remote server
3. Users access the app to get search results and find articles

When 1 user accesses the application, things are fine.

Unfortunately this is an Image by Author

But when 50 users access the application, the single instance gets overloaded!

Image by Author

If we need to upgrade the application, it goes down making searches unavailable.

Image by Author

Some Reasons we Need Kubernetes

There are many more examples, but as you can see deploying an application is only a small portion of the battle. Managing an application is significantly more difficult to do automatically at scale. We need to:

• Scale applications according to workload
• Make changes without application downtime
• Distribute user traffic to multiple instances of our application

Kubernetes is declarative, which means we tell it what we want to happen and Kubernetes does it for us (to a degree). This contrasts with imperative programming like Python, where with every line of code we change the programs state explicitly.

However, you don’t always need k8s. In fact in many cases it’s probably overkill. Check out one of my previous posts on comparing and contrasting different ML deployment strategies.

Kubernetes Objects 101

Kubernetes objects are persistent entities in the Kubernetes ecosystem. They defined how applications run on the cluster. We’ll look at:

• Pods
• Deployments
• Services
• Ingresses

Pods

Pods are the smallest deployable units of computing you can deploy in Kubernetes. They’re typically instances of a containerized application. Basically, running Docker containers.

Image by Author

Deployments

Deployments provide a declarative definition for 1 or more pods, making it easy to scale apps.

Image by Author

Services

Services provide a lookup for pods in a deployment. This is a layer of abstraction over many instances of an application.

Image by Author

Ingresses

Ingresses provide entry into the cluster for requests so that you can expose applications to the world.

Image by Author

Today we’re going to show you how to deploy an application on Kubernetes. You’ll only need to Docker Desktop installed locally with Kubernetes enabled.

Practical

This portion is hands on, use the code samples and gists to follow along.

Setup

1. Install Docker Desktop
2. Enable K8s by going to Preferences -> Kubernetes -> Enable Kubernetes

Testing Our Application with Docker

To deploy an app on Kubernetes, we first need to make sure it works locally with Docker. Use the example fastapi app here: https://github.com/kylegallatin/fast-bad-ml

# clone repo
git clone git@github.com:kylegallatin/fast-bad-ml.git
cd fast-bad-ml
# build and run app with docker 
docker build -t ml-app .
docker run -p 5000:5000 -it ml-app

Ensure you can reach the app at: http://localhost:5000/ and make a prediction at http://localhost:5000/predict?feature_1=0&feature_2=1&feature_3=2.

This is a simple ML application that takes 3 features and returns a binary prediction.

Verify Kubernetes is Working With Docker Desktop

When you enable Kubernetes with Docker Desktop, it installs the Kubernetes CLI kubectl for you and configures it for your local cluster. To ensure it's working, make sure you have kubectl installed.

kubectl version

Now, we can look at some example Kubernetes stuff.

# ensure your using the docker-desktop cluster
kubectl config use-context docker-desktop
# check the "nodes" for your cluster (for docker desktop it's just 1)
kubectl get nodes
# check the namespaces (logical separation of resources)
kubectl get ns 
# check the pods running in a given namespace
kubectl get pods -n kube-system

Cluster Setup

We have 1 quick setup step for our cluster which is to install the nginx ingress controller. This will route traffic to our applications and make URLs available outside the cluster.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml

Verify installation by going to http://localhost:80 in a browser. You should see 404 — Not Found. This looks bad, but actually means things are working as expected!

Image by Author

Write the Object Definitions for our Application

To create Kubernetes resources for your containerized application, you need to write Kubernetes object definitions. Typically this is done using YAML (Yet Another Markup Language) or JSON. We will use YAML to define the resouces we went over today.

Deployment will define the pods:

Service will provide a layer of abstraction over all our pod replicas.

Ingress will let users access our application from outside the cluster:

We can apply these to our cluster directly from the fast-bad-ml repository that was cloned earlier with:

kubectl apply -f deployment.yaml  
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

To verify it’s working, go to http://localhost:80. To make a prediction we can use http://localhost:80/predict?feature_1=0&feature_2=1&feature_3=2. It’ll look like this!

Image by Author

Summary

We just created an app, made a Docker image for it, and deployed that app to Kubernetes on our local machine.

To do this in the cloud, you can use K8s services like EKS (on AWS) or GKE (on GCP) and apply these same Kubernetes objects there to have them run at scale in the cloud. Best of luck!