Quick and Easy Way to Debug SageMaker Pipelines

A brief introduction to AWS SageMaker local mode

Photo by EJ Strat on Unsplash

Introduction

As you may have experienced, debugging a pipeline is time-consuming in a cloud environment. Since our programs interact with cloud-hosted services, we have many points to consider compared to the local environment. For example, imagine your pipeline failed due to a version issue in the last step of the pipeline. Maybe it has been running for 2 hours before failing. In that case, your money on a particular cloud resource is wasted. That’s where Data Scientists/ML Engineers should be aware of cost-effective ways to debug and test their pipelines. In this article, I will guide you on how to debug AWS SageMaker pipelines using their Local Mode feature.

Topics covered in this article.

• What SageMaker pipelines are
• What is SageMaker local mode
• Benefits of using SageMaker local mode
• How to use SageMaker local mode for testing and debugging purpose.

Let’s start.

What is the Sagamaker pipeline?

Example pipeline-illustration — Image by Author

In simple words, the SageMaker pipeline is a directed acyclic graph (DAG) which gives us the capability to define the set of instructions based on our requirements. Each bubble in DAG corresponds to instructions specified by the user. Since these bubbles are interconnected, we can map dependencies among them, which helps us to separate complex instructions into single pieces. In addition, it gives more maintainability to these pipelines.

SageMaker Pipelines Overview

What is SageMaker local mode, and why should we care about it

Before explaining the local mode, we should know how SageMaker pipelines work.

What happens when we execute the SageMaker pipeline?

To explain this, let me take a sample use case with the below steps.

• Data ingestion
• Preprocessing
• Model building
• Inferencing
• Postprocessing

When we execute the above pipeline, SageMaker will automatically spin an EC2 instance for each step and run the script defined by the user. In advance, this whole pipeline will use 5 EC2 instances. And remember, All resources in the cloud have a cost. That’s about the cost factor. How about the time factor? It’s even worse. Except for the script execution time, we must wait a few minutes to spin the EC2 instance. Imagine you are building a pipeline or debugging a pipeline, which means you have to waste unnecessary money and time just on the EC2 instances. What if we can avoid spinning the EC2 instance and run the whole pipeline locally as a pilot run? That’s where SageMaker local mode comes in handy.

With SageMaker local mode enabled, we can try our pipelines on the local machine. This means no more time and money wastage on EC2 instances while we are debugging or testing pipelines.

Let’s build a simple pipeline with preprocessing and training steps. Here I’ll use a custom container. But local mode works with both in-built and custom-defined containers. You can see below my folder structure.

My folder structure — Image by Author

Build preprocessing step

In this step, I’ll use the training dataset inside my data directory and upload them into the preprocessing container. And then execute the python script located in preprocessing directory. Once the processing job is complete, preprocessed files should save to the configured S3 location.

First, we need to set the below session configurations. It is necessary only if you are interacting with the local file system instead of S3.

Now time to define the processing step.

Build training step

This step will perform the below actions.

• Get preprocessing step output and train the model with them.
• Save the model artifacts into the local directory called model

So the question is how to set the estimator output_path as a local directory. We can set local paths by adding afile:// prefix to the local path.

Build the pipeline

Let’s test our pipeline.

When you successfully execute the pipeline, you’ll notice the model.tar.gz file created in the model directory. You can use it for inferencing and model diagnostics etc.

Final Words

If you are using SageMaker for development, you may encounter different bugs and errors. That’s where SageMaker local mode could be your lifesaver. It will enable you to focus more time on your problem. After a local success run, you can ship your model pipeline into production. When converting local mode into AWS managed pipeline, you must replace your local session with a SageMaker session and use the SageMaker IAM role with the necessary permissions.

The complete code is available in my GitHub repo.

Happy learning!!

References:

• AWS Sample repo — https://github.com/aws-samples/amazon-sagemaker-local-mode
• Sagemaker Documentation — https://sagemaker.readthedocs.io/en/stable/overview.html?highlight=local%20mode#local-mode