Multiple Ways to Create Kubernetes Secrets

Tutorial on how to employ Kubernetes secrets for an MLOps pipeline creatively

This article describes how to create Kubernetes (K8s) secrets as part of an installation guide to the Machine Learning Operations (MLOps) pipeline detailed in my post “MLOps on Kubernetes with Docker Desktop”.

Photo by Kristina Flour on Unsplash

The installation in the above-mentioned article uses the following secrets on the K8s pod:

1. github-repo-cred to access the GitHub repo; used as an environment variable or file mount
2. gitlab-registry-cred to push Docker images to GitLab Container Registry (CR); used as an environment variable or file mount
3. gitlab-pull-cred to allow K8s to pull images from GitLab CR; used with imagePullSecrets in the pod definition or added to the service account
4. kubectl-config to perform kubectl operations within the container; used as file mount

In the following, I detail how to create each of them.

1. github-repo-cred holds GitHub credentials and is used to clone a repository from inside the container.

Before the secret actually can be created, base64-coded valid GitHub credentials need to be inserted into the data section for the keys GH_REPO_USER and GH_REPO_TOKEN.

And this is how to generate the base64 code for the user tibfab (that’s me) in the terminal on the Mac.

% echo -n tibfab | base64
dGliZmFi

The secret itself is then created with kubectl apply.

% kubectl apply -f dss-at-k8s/installation/github-repo-cred-secret.yaml
secret/github-repo-cred created

2. gitlab-registry-cred stores GitLab credentials needed for read/write access to GitLab CR to commit and push a running Docker container’s status whenever necessary.

The secret is created similarly to github-repo-cred above, but this time with appropriate GitLab credentials. I used a so-called GitLab deploy token to restrict access to a specific project.

% kubectl apply -f dss-at-k8s/installation/gitlab-registry-cred-secret.yaml
secret/gitlab-registry-cred created

3. gitlab-pull-cred is used by K8s to pull images from GitLab CR upon pod creation.

The easiest way to create the image pull secret is from a Docker login. So if not logged in already, then login to GitLab CR first.

% docker login https://gitlab.com/<user>/<project> -u <user> -p <token>

And create the pull secret with kubectl as follows.

% kubectl create secret generic gitlab-pull-cred \ 
  --from-file=.dockerconfigjson=.docker/config.json \ 
  --type=kubernetes.io/dockerconfigjson
secret/gitlab-pull-cred created

After the pull secret was created, it needs to be referenced by K8s upon pod startup to pull the image from GitLab CR. This can be achieved in two ways.

• Either set imagePullSecrets in the pod configuration

• or add the image pull secret to a service account as described here.

% kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gitlab-pull-cred"}]}'
serviceaccount/default patched

4. kubectl-config is a file mount used to manage K8s from inside the DSS container running on the pod and created as follows:

% kubectl create secret generic kubectl-config --from-file=.kube/config
secret/kubectl-config created

Wrapping up

If all secrets above were created successfully, the list given by kubectl looks like the following:

% kubectl get secrets                                          
NAME                   TYPE                                  DATA   AGE
default-token-2spdb    kubernetes.io/service-account-token   3
github-repo-cred       Opaque                                2
gitlab-pull-cred       kubernetes.io/dockerconfigjson         1
gitlab-registry-cred   Opaque                                2
kubectl-config          Opaque                                1

There are different ways to make credentials available from secrets in the container.

The easiest: Environments defined from a secret in the pod configuration make credentials as environment variables visible in the container. Check out the code snippet below. It sets GL_REG_USER and GL_REG_TOKEN from the secret gitlab-registry-cred.

Another possibility is to volume mount a secret as a file whose name becomes the name of the configuration variable, and it’s content the value of the variable. This has the advantage of being able to apply permission control directly to the pod configuration with the keyword defaultMode. The following code snippet demonstrates that.

An example usage of the credentials is given in a pod configuration below as part of the start command of the container.

Where the command $(cat /github-repo/credentials/GH_REPO_USER) in line 9 returns the content of the secret key GH_REPO_USER and $(cat /github-repo/credentials/GH_REPO_TOKEN) returns the content GH_REPO_TOKEN respectively.

Conclusion

There are multiple ways to create and employ K8s secrets. This post shows how various types of secrets are applied creatively in different parts of an entire MLOps pipeline.