Hands-on Tutorials
Lambda Build and Deployment
There are several ways you can deploy a Lambda function, and the most common way is to simply write your code within the IDE, test, save, then deploy. You may also require it to run at a certain time, again, more point and clicks for Cloudwatch Events. This does not abide by best practice, ease of management such as undoing changes, version control, and code reusability.
Infrastructure as code
Nevertheless, managing and provisioning cloud resources have become a lot easier these days due to the availability of many open-source tools that allows you to define and configure your resources by using a human-readable configuration file such as YAML. Amongst these, AWS Serverless Application Model (SAM) and Serverless, are some of the best frameworks to use to Deploy your Lambda as a dependency, and as an IaC.
Problem Statement: Suspending notebooks resources
Working with many Data Scientists and one of the issues that we often face are notebooks running when they aren’t in use after hours. In this article, I will demonstrate how to use SAM and Serverless to deploy a cron-scheduled Lambda with external dependencies, to suspend SageMaker Notebooks.
The Python code that we will be deploying is as follows:
and the requirements file:
What are we using pandas for… Nothing, I just want to show an example of bringing in external dependencies.
Set up AWS CLI
Make sure AWS CLI is installed on your computer and you’ve set-up the environment with aws configure .
It’s best practise **** to use temporary credentials, or a CI/CD environment for your deployments.
AWS Serverless Application Model (SAM)
Overview of SAM
AWS SAM is an open-source framework that allows you to deploy serverless applications, such as Lambda, to the AWS. It allows for configuration management, local testing and debugging, and integration with DevOps processes. SAM transforms the configuration file into Cloudformation Scripts using best-practice approaches.
Deploying a SAM scheduled Lambda with external dependency
5 Easy steps:
1. Installing SAM CLI
Follow this link to install SAM CLI which we will use to deploy the SAM package onto AWS. SAM requires Docker and an amazon image for its local testing and build deployment since Lambda is a Linux-based environment.
> docker pull amazon/aws-sam-cli-build-image-python3.8
2. Creating your SAM project
Run the command below to create your project:
> sam init --runtime python3.8 --app-template hello-world --name suspend_sm_notebook What you should see are the following folders and files created:
Copy the Python code over to handler.py then you’re ready to construct your YAML file to deploy the Lambda and time-based Cloudwatch event.
3. Constructing the SAM Template
Using SAM, you can deploy various serverless models but since we want to deploy Lambda, the resource type will be Function. A policy was also added which allows for its interaction with SageMaker notebooks. SAM Event will invoke the Lambda function via the Schedule through Cron expressions.
4. Building your SAM
Once you’ve tested, you can build your deployment which will package your code with any dependencies from requirements. Due to the nature of pandas it must be compiled in a lambda-like container.
`> sam build –use-container –manifest source/requirements.txt`
5. Deploying your SAM
Finally, you can deploy onto your AWS environment. SAM will use the AWS config that is set up on your device, just make sure it has the access required and an existing S3 bucket.
`> sam deploy –stack-name SuspendSMNotebookSAM –s3-bucket sam-lambda-deploy-apse2-jy –region ap-southeast-2 –capabilities CAPABILITY_IAM`
Serverless Application Framework
Overview of Serverless
Serverless is an open-source framework that allows you to deploy serverless applications to the cloud. Unlike SAM, it also allows you to deploy to other clouds like GCP and Azure, and has a free hosted dashboard to provide a full application lifecycle management view.
Deploying a SAM scheduled Lambda with external dependency
4 Easy steps:
1. Installing Serverless CLI
Follow this link to get set-up with Serverless. For those who have node installed already, you can install using npm. Serverless allows you to install plugins which can be used for deployment. In this case, the python-requirements plugin was used to allow Serverless to package python with the dependencies in requirements.
`> npm install -g serverless`
`> docker pull lambci/lambda:build-python3.8`
`> serverless plugin install -n serverless-python-requirements`
2. Creating your Serverless project
Run the command below to create your project:
`> serverless create –template aws-python3 –path suspend_sm_notebook_serverless`
What you should see are the following files created:
3. Constructing the Serverless Template
As a multi-cloud framework, you must define the target provider as AWS and ap-southeast-2 region. The policy and role for Lambda were added to grant access to SageMaker, and schedule to cron-scheduled invocations.
4. Deploying your Serverless
This step is straightforward, simply run the command below and it will convert your Serverless template to Cloudformation in the background, package dependencies in requirements, and create an S3 bucket for your deployment.
> serverless deploy
Rollbacks and removing deployments (Optional)
Serverless allows you to rollback to a previous deployment:
> serverless deploy list
> serverless rollback --timestamp <timestamp>
or clean-up and remove our deployed stack from AWS.
> serverless remove
So… Which one should I use?
As you saw, both AWS SAM and Serverless completes the goal of allowing you to deploy a cron-scheduled Lambda to AWS with external platform-dependent packages. The frameworks are also similar in nature in both configuration format and simplicity. However, they do have their strengths and weaknesses:
Some factors when deciding between Serverless and SAM should be based on what you’re trying to achieve, your team capabilities and expertise, and a strong community (being open-sourced).
As a recommendation, if you work in a multi-cloud environment, Serverless may be a better option, but if local testing and evaluation are more important, then SAM may fit your team better.
About Me
I love writing medium articles and contribute towards a knowledge-sharing community. Outside trying out new food recipes, I help businesses build cloud and data solutions. Feel free to connect with me and say hi!
