The world’s leading publication for data science, AI, and ML professionals.

Overcoming AWS CodePipeline’s Design Flaws to Automate Serverless ML Architecture Deployment

Build a simple CI/CD pipeline that automatically creates deploys a lambda function and associated SQS queues on Amazon within 15 minutes.

Photo by eberhard grossgasteiger from Pexels
Photo by eberhard grossgasteiger from Pexels

Serverless applications are needed for machine learning and data science. They are lightweight, pay-as-you-go functions that reduce time spent on setting up servers or instances for your machine learning infrastructure. AWS has made it easier to construct a CI/CD pipeline with CodeCommit, CodeBuild, CodeDeploy, and CodePipeline. Data scientists can spend less time on cloud architecture and DevOps, and spend more time fine-tuning their models/analyzing data. Plus, the pay-as-you-go model is cheaper than paying for cloud servers/EC2 instances to run 24/7 just to host the infrastructure.

In this article, we’ll go over using AWS native tools to construct a CI/CD pipeline for deploying a simple machine learning model as a serverless endpoint. This pipeline (using CodePipeline) reads all code from a CodeCommit repository and triggers a CodeBuild. This CodeBuild uses Serverless Framework (a software package for YAML + CLI development) to deploy the code in CodeCommit to AWS Lambda Function. The goal is to trigger an AWS Lambda deployment whenever we make changes to a CodeCommit repository.

Before We Begin

  • This article will cover a lot of concepts, that may be unfamiliar to many of you. Please view the Terminology and Suggested Reading to understand more details.
  • DISCLAIMER: Because AWS Lambda is lightweight, it may not be appropriate for huge, deep learning models. See Why We Don’t Use Lambda for Serverless Machine Learning for more information.
  • AWS Lambda and Serverless are normally used interchangeably. To avoid confusion, I’ll refer to AWS Lambda when talking about the pay-as-you-go architecture. I’ll refer to Serverless Framework when talking about the CLI tool used to deploy to Lambda.

Terminology

  • AWS SQS – Simple Queue Service. A fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
  • AWS SageMaker – helps data scientists and developers to prepare, build, train, and deploy high-quality machine learning (ML) models quickly by bringing together a broad set of capabilities purpose-built for ML. Used to deploy ML models separately from AWS Lambda.
  • AWS Lambda – a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes.
  • Serverless Framework – a software package for YAML + CLI development and deployment to AWS Lambda.
  • AWS CloudWatch – A tool for monitoring and observability. Keeps track of Lambda output in logs.

Suggested Reading

To understand more on AWS Lambda, I highly recommend reading Serverless Functions and Using AWS Lambda with S3 Buckets.

To understand more about deploying in the Serverless Framework, see the Serverless Framework Documentation.

To understand CI/CD and CodePipeline development in AWS, I highly recommend reading CI/CD- Logical and Practical Approach to Build Four Step pipeline on AWS.

To understand more about SageMaker and SQS, see the documentation on AWS.

Architecture

Serverless architecture diagram I created using Codecraft
Serverless architecture diagram I created using Codecraft

The following steps are as follows

  • The developer deploys a model on SageMaker
  • The developer updates via git the CodeCommit repository that contains logic for the architecture
  • CodePipeline notices a change in CodeCommit repository via CloudWatch. It then triggers CodeBuild to create/update, link, and deploy all 4 services (3 SQS queues and 1 lambda function) in AWS.
  • The developer then sends a message from TriggerLambdaSQS. This TriggerLambdaSQS then triggers the ModelPredictLambda function with input data. The Lambda will use the model deployed from SageMaker. If there is an error in the message format/lambda, the failed message will be outputted to the DeadletterSQS. If the lambda function processes the message successfully, it will output the result of SageMaker to the CloudWatch Logs.

Why use CodeBuild for serverless deployment? Doesn’t AWS CodeDeploy have a built-in Lambda deployment?

Correct, CodeDeploy allows lambda deployment. For every commit to CodeCommit, you can manually call a CodeBuild and then a CodeDeploy. We want to automate the build and deployment stages via CodePipeline, and not worry about manual triggers.

Ok, but why not use CodeDeploy in CodePipeline?

In CodeDeploy, deploying packages on lambda functions is different than deploying packages on EC2 instances. In deploying on a lambda function, CodeBuild takes only the YAML file that lists lambda deployment configurations and stores it in an S3 Bucket. CodeDeploy downloads the YAML file and deploys it on a lambda function. In deploying on an EC2, CodeBuild takes all the files from CodeCommit, zips them, and stores the zip file in an S3 Bucket. CodeDeploy downloads and unzips the file on an EC2. CodePipeline specifies that artifacts generated from CodeBuild MUST be in zip format for them to be passed to CodeDeploy. This makes automated lambda deployment tricky, as it depends on a YAML file for deployment. This is a known problem in AWS discussion forums and Amazon did promise to deliver a fix for this bug. As of 2021, there still hasn’t been an update.

The workaround is to create a CodePipeline that uses both CodeCommit and CodeBuild. In CodeBuild, you install all packages (including the serverless framework) and run the serverless framework to deploy the application to AWS Lambda. Since you’re relying on the serverless framework package instead of CodeDeploy for deployment, you’ll need to create a different YAML file called serverless.yml (no need for appspec.yml, a default YAML file for CodeDeployment). We’ll discuss more later on.

Tutorial

For this tutorial, we’ll use scikit-learn’s Boston home prices dataset. We’ll train an XGBoost model to predict the median home price in Boston. We’ll deploy the model on SageMaker, and use it for predictions in the AWS Lambda architecture.

Create Folder Structure

We’ll first want to create a folder called repo. This will contain a folder (lambda_deployment) and a Python script (boston_sagemaker_deployment.py) on the same level. lambda_deployment will contain all the relevant files needed to successfully deploy a lambda function in CodePipeline. Below is a folder structure of the files.

Screenshot of folder structure for REPO in Visual Studio Code
Screenshot of folder structure for REPO in Visual Studio Code
  • buildspec.yml – a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build and deploy using the serverless framework
  • model_predict_lambda.py – the lambda function to call the SageMaker model for predictions
  • requirements.txt – list of python packages to download from pip
  • serverless.yml – configurations in YAML to define a serverless service. This also includes additional components (SQS) that are needed to be deployed along with the lambda
  • boston_sagemaker_deployment.py – script to build and deploy a model on AWS using SageMaker

Create and Execute SageMaker Script

First thing is to deploy a simple XGBoost Model to AWS. Note: we’ll want to create an IAM role called sagemaker-role, with full permissions to access SageMaker. I won’t go into any detail about XGBoost, as it is outside the scope of this tutorial.

After that, we’ll execute the script.

python boston_sagemaker_deployment.py

We only have to do this once. It’ll take a few minutes, but it’ll give a SUCCESS or FAILED message. If SUCCESS, it’ll return an endpoint for our model. Check on the SageMaker AWS console that the endpoint is there.

Create and Upload lambda_deployment Folder and Files to CodeCommit

CodeCommit is Amazon’s source control service that hosts secure Git-based repositories. We’ll upload the lambda_deployment folder there. We can either use git to add the files or add them manually via the console.

Screenshot of AWS CodeCommit with the uploaded files in lambda deployment
Screenshot of AWS CodeCommit with the uploaded files in lambda deployment

_UPDATE: CodeBuild only recognizes files from the head of repo, not from the head of repo/lambda_deployment. To work around this, move all 4 files in lambda_deployment up to the root of repo. Once those are moved, delete the rest of the files/folders._

Add Files for CodeBuild Environment

CodeBuild allows us to build and test code with continuous scaling. As mentioned before, we’ll also use the serverless framework to deploy the lambda function and services in CodeBuild.

We’ll add in buildspec.yml, which contains configurations for CodeBuild. Since we’re using a clean environment/image, we want to specify the commands needed to download certain packages. This is assuming a clean Ubuntu environment:

version: 0.2
phases:
  pre_build:
    commands:
      - echo "Running pre build commands"
      - apt-get update
      - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7EA0A9C3F273FCD8
      - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
      - apt-get -y install docker-ce docker-ce-cli containerd.io
      - npm install -g serverless --unsafe
      - npm i -D serverless-dotenv-plugin
      - npm install serverless-plugin-aws-alerts --save-dev
      - sls plugin install -n serverless-python-requirements
      - apt-get -y install python3-pip
      - pip3 install awscli --upgrade --user
      - pip3 install -r requirements.txt
      - echo "Reducing size of SageMaker to run on lambda"
      - pip3 install sagemaker --target sagemaker-installation
      - cd sagemaker-installation
      - find . -type d -name "tests" -exec rm -rfv {} +
      - find . -type d -name "__pycache__" -exec rm -rfv {} +
      - zip -r sagemaker_lambda_light.zip
      - cd ..
  build:
    commands:.
      - echo "Running build commands"
      - sls deploy --verbose
      - echo "Finished deploying to Lambda and SQS"

Because it is a clean image, we are responsible for installing pip, awscli, serverless, docker, etc. Note: lambda has a limit to how much we can install on the function. SageMaker has a lot of dependencies, so we’re removing the components of SageMaker that are not relevant for deployment.

The prebuild commands focus on installing the packages, including the pip3 packages in requirements.txt (shown below):

boto3

Now that we have the CodeBuild commands, we next want to set up serverless.yml. Serverless.yml contains configurations to deploy the lambda function and associated services to trigger the function, store the function outputs, send alarms on threshold limits, log function behavior, etc. The file will look like this.

service: model-predict
package:
  include:
    - ./model_predict_lambda.py
provider:
  name: aws
  runtime: python3.6
  iamRoleStatements:
    - Effect: "Allow"
      Action: 
        - "lambda:InvokeFunction"
        - "iam:GetRole"
        - "sqs:CreateQueue"
        - "sqs:GetQueueUrl"
        - "sqs:SendMessage"
        - "ecr:DescribeRepositories"        
        - "cloudformation:DescribeStacks"
        - "sagemaker:InvokeEndpoint" 
      Resource: "*"
functions:
  get_model_predictions:
    handler: model_predict_lambda.model_predict_handler
    provisionedConcurrency: 1 # optional, Count of provisioned lambda instances
    reservedConcurrency: 1 # optional, reserved concurrency limit for this function. By default, AWS uses account concurrency limit
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - TriggerLambdaSQS
              - Arn
          batchSize: 1
resources:
  Resources:
    TriggerLambdaSQS:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "TriggerLambdaSQS"
        VisibilityTimeout: 30
        MessageRetentionPeriod: 60
        RedrivePolicy:
          deadLetterTargetArn:
            "Fn::GetAtt":
              - DeadletterSQS
              - Arn
          maxReceiveCount: 1
    DeadletterSQS:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "DeadletterSQS"
        VisibilityTimeout: 30
        MessageRetentionPeriod: 60
plugins:
  - serverless-python-requirements

Here’s a brief overview of what serverless framework is creating

  • a service called model-predict, which will encapsulate the lambda function and its resources
  • a package of all python files required for the lambda function (in this case, _model_predictlambda.py)
  • an AWS provider and permissions assigned to the IAM role created for this function
  • a lambda function called _get_modelpredictions, which includes a handler that points to the _model_predicthandler function in _model_predictlambda.py. It also contains concurrency limits and a reference to the SQS queue listed in resources
  • a resource section that contains TriggerLambdaSQS and DeadletterSQS. TriggerLambdaSQS is where we send the messages to the lambda function. DeadletterSQS contains all the messages that failed when being processed in lambda. A dead letter queue is added to make sure that TriggerLambdaSQS doesn’t hold onto the failed messages (which can be resent to the lambda function and trigger a continuous loop of failed messages, resulting in a lot of failed lambda invocations and a surge in prices)

Now, we want to create the python file to store the lambda function.

Because we’re using an SQS to trigger a Lambda function, we’re dealing with a queue of asynchronous events. Hence, we’re traversing through the "Records" parameter to process a list of events in the queue at a current moment. For now, we’re going to print out the result and return a successful message (which will be deleted immediately as we set the batchSize in serverless.yml to 1).

Create CodeBuild

Now that we uploaded repo to CodeCommitt, let’s create a CodeBuild. Navigate to Developer Tools -> CodeBuild -> Build Projects.

Screenshot of AWS CodeBuild dashboard
Screenshot of AWS CodeBuild dashboard

From there, click on the orange button, Create build project.

Screenshot of Create build project -> Project configuration
Screenshot of Create build project -> Project configuration

Let’s name the project serverless-build. We then scroll down to initiate the source of the CodeBuild.

Screenshot of Create build project → Source
Screenshot of Create build project → Source

Select the appropriate branch where the code is.

Next, we’ll configure the environment for the build.

Screenshot of Create build project →Environment
Screenshot of Create build project →Environment

We have the choice of using a custom Docker image on Amazon ECS or creating a new image. I recommend using Ubuntu, as it is compatible with a lot of machine learning libraries. Because we haven’t created a service role for the CodeBuild earlier, we’ll create a new service role for the account.

We’ll then specify the buidspec file, batch configurations, and artifacts.

Screenshot of Create build project → Buidspec, Batch configuration, Artifacts
Screenshot of Create build project → Buidspec, Batch configuration, Artifacts

Buildspec, we already defined the buildspec.yml file in repo. So we just point to that file. Artifacts are zip files stored in S3 that are outputted from CodeBuild and sent to other stages in the pipeline. Since we are just deploying using serverless in the build step, we can leave this empty.

Next, we’ll instantiate CloudWatch logs to monitor the CodeBuild process.

Screenshot of Create build project → Logs
Screenshot of Create build project → Logs

We’ll leave everything blank, and check the CloudWatch logs checkbox. Finally, we click on Create build project.

Screenshot of server-build in Build projects dashboard
Screenshot of server-build in Build projects dashboard

So our build project is created. We can click on Start Build orange button, but we can let CodePipeline handle that.

Add Permissions to CodeBuild Service Role

Remember how we created a new service role in CodeBuild titled codebuild-serverless-build-service-role? While this creates a new service role with that name, we still need to add permissions to that role so that it can access other AWS components in our architecture (Lambda, SQS, CloudFormation, CloudWatch logs, etc). Before creating the CodePipeline, please check that the following permissions added in the service role.

  • AmazonSQSFullAccess
  • IAMFullAccess
  • AmazonS3FullAccess
  • CloudWatchLogFullAccess
  • AWSLambdaSQSQueueExecutionRole
  • AWSCloudFormationFullAccess
  • AWSLambda_FullAccess
Screenshot of policies and permissions added tor codebuild-serverless-build-service-role
Screenshot of policies and permissions added tor codebuild-serverless-build-service-role

Create CodePipeline

CodePipeline watches repo in CodeCommit. It triggers a pipeline whenever we add a commit to repo. This is convenient because we automated a deployment whenever we make changes. Navigate to Developer Tools -> CodePipeline -> Pipeline.

Screenshot of CodePipeline dashboard
Screenshot of CodePipeline dashboard

Click on the orange button Create pipeline.

Screenshot of Create new pipeline → Choose pipeline settings
Screenshot of Create new pipeline → Choose pipeline settings

We now can configure settings for the pipeline. In Step 1, name the pipeline serverless-pipeline. Because we haven’t created a service role for the CodePipeline earlier, we’ll create a new service role for the account. Use default values for everything else. Use the default values for Advanced Settings and click Next.

Screenshot of Create new pipeline → Add source stage
Screenshot of Create new pipeline → Add source stage

We now add the source stage of the pipeline, which points to repo in CodeCommit. Use default values and branch of choosing, and click on Next.

Screenshot of Create new pipeline → Add build stage
Screenshot of Create new pipeline → Add build stage

We now add the build stage of the pipeline. We set the build provider to CodeBuild and the project name to serverless-build (which is what we created earlier). We do a single build and click Next.

Screenshot of Create new pipeline → Add deploy stage
Screenshot of Create new pipeline → Add deploy stage

Because we didn’t add a CodeDeploy stage (and because we’re doing serverless deployment in CodeBuild), we can skip this step. Click on Skip deploy stage.

Screenshot of Create new pipeline → Review
Screenshot of Create new pipeline → Review

This is just to review everything before creating the pipeline. It looks good, so let’s click on Create pipeline.

Screenshot of Pipeline → serverless-pipeline
Screenshot of Pipeline → serverless-pipeline

Our CodePipeline is now completed. This has the Source and Build stage. Source stage is blue because it is in progress. It will be green if it is successful, and red if it failed. Once both stages are green, we can begin testing our architecture.

Testing architecture

If everything builds correctly, we should see Lambda -> Function-> model-predict-dev-get_model_predictions created. We should also see two queues created (DeadletterSQS and TriggerLambdaSQS).

Screenshot of Amazon SQS dashboard, where queues created are from the serverless.yaml
Screenshot of Amazon SQS dashboard, where queues created are from the serverless.yaml

If we click on TriggerLambdaSQS -> Send and receive messages, we’ll see a message we can send to the lambda function. Let’s send a JSON format to pass into the model.

Screenshot of TriggerLambdaSQS → Send and receive messages
Screenshot of TriggerLambdaSQS → Send and receive messages

This is what we send in the Message Body.

{ "payload": "14.0507,0,18.1,0,0.597,6.657,100,1.5275,24,666,20.2,35.05,21.22"}

After clicking on Send Message, we navigate to CloudWatch -> Log Groups -> /aws/lambda/model-predict-dev-get_model_predictions. We get the latest time stamp and inspect the logs.

Screenshot of CloudWatch logs, depicting the output of lambda
Screenshot of CloudWatch logs, depicting the output of lambda

We can see the output 75.875, which represents the Boston Housing mean sample price, based on the input we sent.

Conclusion

We went over how to construct a CI/CD pipeline in AWS to deploy a lambda architecture. The architecture contains SQS queues for trigger/failed messages and CloudWatch to output the results to logs. We went over a workaround for creating a serverless pipeline by utilizing CodeBuild and serverless framework for deployment.

We also created a script to utilize SageMaker for hosting the XGBoost model. We referred to the SageMaker endpoint in our lambda function and got the prediction of the Boston Housing median price with input fed from SQS. We are able to see the prediction in CloudWatch logs.

Aws Codepipeline has made it easier for data scientists to perform MLOps. Hopefully, Amazon can fix the bug in CodePipeline to allow YAML files to be passed in between CodeBuild and CodeDeploy stages within the pipeline.

Note: Make sure to delete SageMaker endpoint when you’re done

hd2zm/Data-Science-Projects


Thanks for reading! If you want to read more of my work, view my Table of Contents.

If you’re not a Medium paid member, but are interested in subscribing to Towards Data Science just to read tutorials and articles like this, click here to enroll in a membership. Enrolling in this link means I get paid for referring you to Medium.


Related Articles