Monitor Your AWS Lambda Functions (or Any Other AWS Service) in Slack

As organizations develop, it is common to have ELT or batch inference jobs living in AWS Lambda that run periodically. There are many articles out there on monitoring Lambda errors using CloudWatch Alarms, but for periodic lambda jobs, this is both overly complex and doesn’t provide much useful information in the notification other than that there was an error.

Here I propose a more direct way of monitoring those periodic Lambda functions. With this method, you will get a notification containing the error message and traceback for each error directly in your slack channel, eliminating the need to dig into the logs to troubleshoot the error. While this method does involve CloudWatch, it does not require CloudWatch Alarms or any message queuing Service like SQS or Kenisis.

A quick warning before we begin: If your Lambda function processes many requests per minute and begins to fail, your slack channel will blow up 💥 with error messages. High-volume lambda functions are better off using one of the methods described in detail in the following resources:

• Tracking AWS Lambda Functions error via Slack
• Slack Notifications from AWS CloudWatch Alarms
• AWS Lambda Blueprint (2015)

Overview of this approach

See the basic architecture below.

The user sends a request to Api Gateway, which forwards that request to a model we have deployed with Lambda. That Lambda model will return a prediction or an error to the API Gateway to send back to the user and will also write a log message to Cloud Watch logs.

So far, we have a standard model deployment on AWS Lambda. The magic happens when CloudWatch triggers our Notification Lambda to send a message to Slack whenever the log contains the text ERROR.

How we’ll implement it:

1. Deploy a simple linear model to a lambda function and connect it to an API gateway endpoint.
2. Create a Slack Webhook.
3. Create a Lambda function to send messages to Slack.
4. Configure a CloudWatch to trigger our notification Lambda.

Let’s dive in:

As a test case, we will deploy a simple model that doubles a number and raises a ValueError if the input is not a number. If you already have a lambda function or other AWS service you want to monitor, you can jump ahead to Create a Slack Webhook.

Create a new Lambda function:

1. Go to the Lambda service in the AWS console.
2. Select Functions from the sidebar.
3. Select Create function to – you guessed it – create a new function.

4. Choose Author from Scratch, enter a name for your Lambda function, and select the runtime. I will call my function MonitorMe, and I am using Python 3.8 as the runtime. Accept the rest of the default settings and click Create function.

5. Use the following code for our simple Lambda model.

   Stepping through the code:

API Gateway forwards query string parameters to our Lambda function through the event parameter.

The above code first sets the variable x to event['queryStringParameters']['x']. Every element in event['queryStringParameters'] is a string, so to use x for any numeric calculations, we must cast it to a numeric data type. We do that with x = float(x). floatis a data type representing a floating-point number, so attempting to cast anything that is not a number to floatwill cause our Lambda to throw an exception. Finally, we double x and return it in the body of our response.

Create the API Gateway Endpoint

Notifications would work without this step, but setting up an API is relatively straightforward and provides our MonitorMe Lambda with a realistic event variable to process.

1. Visit the API gateway service and select Build on the HTTP API panel.

2. Connect our MonitorMe Lambda function as an API integration.

Click Add integration

• Select Lambda as the integration type
• Make sure the correct region is selected (this should match the AWS region your Lambda is hosted in,) then search for the MonitorMe Lambda function we created above.
• Provide a name for the API. I’m calling my API MyAPI. (clever, I know)
• Click Next

3. Now we need a route. A route assigns an action to a URL path. The action we are going to set up forwards a request to our MonitorMeLambda function.

• Enter a logical path for the Resource path. I am using the name of our Lambda function MonitorMe, but you can use anything
• Select MonitorMe as the Integration target.
• Click Next.

4. On the next screen, accept the defaults.

Make sure Auto-deploy is turned on. Without it, we have to deploy any changes to our API manually. This would be useful if we needed to deploy to a development API for testing before deploying into the production API. We want to keep things simple for this example.

The API is deployed. Let’s Test it!

1. Find the Invoke URL on the MyAPI page for the $default stage. This is our base URL. Copy it.

2. Add the route path we created in API creation step 3 to the base URL. Replace {YOUR API BASE URL} with base URL you copied in the previous step. (The URL is case insensitive, but I’ve capitalized it for clarity):

https://{YOUR API BASE URL}/MonitorMe

3. Add the query to the URL by appending a ?x= followed by the value to test. This is where we send parameters to the MonitorMe Lambda function. For example:

https://{YOUR API BASE URL}/MonitorMe?x=2

4. Plug this into your browser, and vualá. 2×2= 4.0

That’s one super useful model. 😉

Create a Slack webhook

At last, we start to get into the meat of the project. We will send messages to Slack by way of a webhook.

1. Create a channel for your notifications. I’m calling mine aws-notifications

2. Visit api.slack.com/apps. If you are not signed in, you will see a message asking you to sign in to your Slack account to create an application. Do it.

3. Once you are signed in, you will instead see a button labeled Create an App. Click it.

4. We want to create our app from scratch.

5. On the next screen, add an app name, select your workspace, and click Create App.

6. Once you create the app, the Slack API website conveniently suggests some features. We want Incoming Webhooks, so find that box and click it.

7. On the incoming webhooks page, click the slider in the upper right corner to turn webhooks on. This will also expand a settings section at the bottom of the page.

8. Click the Add New Webhook to Workspacebutton in the settings section at the bottom of the page.

9. Select the channel we created earlier and Allow.

10. Now, you will see that you have created a webhook URL. The curl example updates so we can test it out at the command line.

Simply grab the Sample curl request to post to channelcode and paste it into your terminal like so (If you are on an older version of Windows, you can install cURL by following the directions in this StackOverflow answer):

It will return ok. Check your slack channel, and you will see a post from AWS Notifier that says Hello, World!.

Yay! We can now post to our slack channel.

11. Copy the Webhook URL. We are going to need it later.

Create a Lambda function to send messages to Slack

In this section, we will set up a Lambda function that, when triggered, will send us the Slack message with the error details.

Just like before, create a new Lambda function:

1. Go to the Lambda service in the AWS console.
2. Select Functions from the sidebar.
3. Select Create function to – you guessed it again – create a new function.

4. Choose Author from Scratch, enter a name for your function, and then select the runtime. For this step, I will call the function Notifyand use Python 3.8 as the runtime. Accept the rest of the default settings and click Create function.

5. Copy and paste the following code into your new Lambda function.

   Let’s step through the code

• Set SLACK_WEBHOOK to the slack webhook URL you created earlier. You must update this to the URL of the webhook you created.
• AWS calls the lambda_handler function when the Lambda is invoked. AWS passes in eventandcontext as parameters. event contains the encoded error message.

• Next, Lambda callsdecode_logpayload to decode the error message. This function does three things:

  1. Use base64decode to decode the error string.
  2. Use gzip to decompress the decoded error string.
  3. Convert the resulting JSON to a Python dictionary and return it.

• Then we build the slack message. Details on Slack message construction are available in [Creating rich message layouts](http://Creating rich message layouts) in the Slack API documentation. • We add each logEvent to an attachments list. This ensures we get the entire log message. • Then, we build the message with the main text equal to the log group name, which identifies the Lambda that threw the error, and the list of logEvents, which contains the error log.

• Finally, we create a request object with urllib’sRequest class and send it to the SLACK_ENDPOINT with’s geturl method.

Last step: Configure a CloudWatch trigger for our notification Lambda.

There is one detail here that is very important. We create the trigger on Notify Lambda, not the Lambda throwing the error. This means that if you need to monitor several lambda functions, you add one trigger for each to the Notify Lambda. Fortunately, AWS does not impose a limit on the number of triggers for a single Lambda function.

Setting up the trigger

1. While still in our Notify Lambda, select Add trigger from the top of the page.

2. This will bring up the Trigger configuration dialog. • Select CloudWatch Logs as the trigger type. • Select the Log group corresponding to the Lambda’s logs you want to monitor. If you are following along, this will be /aws/lambda/MonitorMe. • Name the filter. This can be anything. Go wild. • The filter pattern will determine which logs trigger the notification Lambda. Here I will simply enter ERROR which will match any logs that contain the text ERROR.

You can find more information about CloudWatch Filters on the Filter and pattern syntax page in the CloudWatch Documentation.

• Click Add in the lower right of the dialog.

And we’re done! 🌴

Testing

Let’s make sure this all works as expected. We have two cases to test:

1. We should not get a notification in Slack if the MonitorMe Lambda logs a message that is not an error.
2. We should get a useful message if the MonitorMe Lambda does log an error.

Case 1: Successful MonitorMe execution

To test this, we hit our API endpoint with the same URL we used to test the endpoint initially:

https://{YOUR API BASE URL}/MonitorMe?x=2

And we verify that no messages were sent to Slack:

Nothing happened. Success!

Case 2: Unsuccessful MonitorMe execution

Let’s modify the API call to send a string instead of a number. This will cause the Lambda to throw an error.

https://{YOUR API BASE URL}/MonitorMe?x=wompwomp

Excellent. An error. Let’s look at Slack.

Amazing! Not only did we get a notification in Slack, the notification came complete with the error traceback! We can see that it was line 7 of lambda_function.pyand that the error comes from trying to cast wompwomp to float.

Conclusion

There are other ways to capture and send errors to Slack from any AWS service. I listed some articles at the beginning of this post, but when you expect a low volume of errors, this approach is superior because it is more straightforward and gives you the entire error message right in Slack.

At Komaza, we use this approach to monitor pipelines in proof-of-concept projects, data quality checks, and some Lambda functions manually triggered by one of our internal web applications. We’ve caught errors like database schema changes, expired tokens, and various other issues immediately and were able to deploy fixes in under 24 hours in all cases. This relatively simple approach to Lambda monitoring has improved our stakeholder experience by ensuring higher quality data and helping us respond faster to system failures.

Komaza is hiring

If you want to help us provide life-changing economic opportunities for small-holder farmers while powering one of the planet’s most effective carbon sequestration engines, check out our careers page. We are growing fast and have data teams in Kenya and the US.

Now go do some good.

---

References

[1] AWS Architecture Icons & Slack Media Kit

Other useful articles:

• Creating Rich Message Layouts (Slack Documentation)
• AWS CloudWatch logs – Filter and Pattern Syntax (AWS Documentation)
• Setting up a Slack Webhook for simple Notifications by Bradley Schoeneweis
• Tracking AWS Lambda Functions error via Slack by Femi Oladeji
• Slack Notifications from AWS CloudWatch Alarms (Not attributed)
• AWS Lambda Blueprint (2015, AWS Documentation)
• AWS CloudWatch logs – Filter and Pattern Syntax (AWS Documentation)