Go, Neo4J, and AWS Lambda….
And the nitty-gritty of making them work together
I work with Neo4j almost every day and being a Python developer, I mostly couple the both of them together for writing most of my logic. Over the past few days, however, since I started using Golang, I seem to have developed a strong affinity towards it.
I wanted to build a simple service using AWS Lambda that would execute simple queries against my Neo4j Database and decided to use Go for a change. Knowing that there are official Go drivers for Neo4j was an obvious relief. But the first few attempts turned out to be utter failures.
Don’t worry though, the good news is, you can definitely use AWS Lambda, Go and Neo4j together. There’s just a few extra steps that need to be carried out first.
The Catch…
Yes, Neo4j has an official driver for Golang.
No, it's not written in pure Go.
What does this mean? The neo4j-go-driver depends on a C library called “Seabolt” for connecting to the database instance over the bolt protocol.
This means you’ll have to explicitly install the library before you can start executing the Go code for Neo4j. Moreover, seabolt in turn requires OpenSSL (TLS in case of Windows) to be installed on your local system for it to function well.
Installing OpenSSL and Seabolt on your local machine is pretty hassle-free. In fact, OpenSSL comes pre-installed in many Operating systems. (I used a Linux system which already had OpenSSL and all I had to do was install Seabolt after following these instructions)
The challenge arises when we want to replicate the same thing using AWS Lambda. Go, being a statically typed language is first compiled into a binary file which is then made available as a handler for the Lambda function. Typically this won’t cause any problems if all the dependencies are Go-based or if you’re working on your local system.
However, In our case, the Go binary depends on external C object files and links them at runtime. But How do we make these C files available to the lambda functions?
The Solution
After spending a considerable amount of time scouring the internet for a possible solution, I couldn’t come across anything concrete, especially for Go. Most solutions are targeted towards python (since certain python libraries use C dependencies too) and I was almost tempted to go back to my old friend Python. But, even if the languages are different, the concept remains the same.
Enter AWS Lambda Layers….
In the simplest sense, a Lambda Layer is a piece of code that can be shared by different services. Typically, a layer will consist of a bunch of heavyweight dependencies that are required by most of the functions. In our case, we will package our C dependencies together and make them available to the lambda function using Layers.
TL;DR
The entire code including an example lambda function and instructions for setup is available on my GitHub repository.
Step 1: Packaging the C libraries
Since AWS often uses CentOS-based containers for deploying lambda functions, Docker becomes an obvious choice to download and compile the C dependencies.
We create a custom docker image using lambci/lambda-base image as the base image and install all our dependencies on it.
The base image already has OpenSSL installed and configured. The only library needed now is Seabolt.
Seabolt is downloaded as a .tar package and extracted to the /usr/local/lib64 directory within the docker image. To make it available to the Lambda functions, we move the contents to the /opt/lib directory which is one the directories that are searched during runtime linking. ( This is specified in an environment variable called LD_LIBRARY_PATH)
Once the image is built, we need to dump the contents of the /opt/lib folder to a folder on our local system that will be later converted to a layer
Running this script will create a directory called layer which will contain the .so files specific to Seabolt within the layer/lib folder.
Step 2: Create A Layer
Now, all we have to do is create a Layer. There are two ways of doing this. First, we create a Layer using the AWS console or the CLI and upload the contents of the layer/ directory as a .zip file.
Another way is to use the serverless framework that will do all the work for you. All you need is a file call serverless.yml that specifies the details for your layer.
And run simply run serverless deploy from your working directory.
You will get ARN URL for your newly created layer which you can easily attach to your lambda functions.
Next Steps: Create your Own Go Handler Function
Once the layer is successfully created, you can go ahead and create your custom handler functions that can communicate with Neo4J.
The GitHub repository includes a sample handler function that creates a simple node in Neo4j which is deployed using a serverless framework. For more reference on how to create Lambda functions using Go and serverless, this is a useful link to get started with.
Thanks for reading!
