Import Python Libraries for AWS Lambda Layers

Are you getting libraries or dependencies missing errors when you work with your lambda function code with python? Do you want to keep your lambda deployment package smaller and cleaner? Here you are in the right place…

Overview of AWS Lambda Layers

AWS Lambda Layer is simply archived files that contain additional libraries or other dependencies that you can use at runtime for your lambda workloads. That is very useful if you have multiple Lambda functions that use the same set of libraries or dependencies and improve the reusability of packages.

When you add a lambda layer into a lambda function, the contents of the archived files are extracted to the /opt directory in the execution environment. You can have up to five layers per function and it increases the lambda deployment size limits. The lambda layers are deployed as a fixed version and the version number gets increments each time when you add a new layer. When you add a layer to a lambda function, you have to specify the layer version and that will help you to backward compatibility with older versions of dependencies.

Benefits of AWS Lambda Layers

• Make your workload modular and easily deploy.
• By default, lambda layers are enabled with versioning and allow you to use previous versions when needed.
• Since the layer can be used across multiple lambda function packages re-usability is there.
• Very easy to do the changes quickly via the AWS lambda console.

Let’s get started!

In this post, I’m going to explain how to locally build and import external python libraries (pandas & pretty_html_table) and add it to lambda layers in your own lambda workload.

Unable to import module ‘lambda_function’?

In a normal way, if you try to import pandas python library directly into your lambda code, you will get an error as follows.

The problem here is there is no way to install python modules like you are using in your local machine using pip commands. Ok, then the next question is how can I add these libraries to the AWS Lambda?

Here is the Solution

Prepare the Required Packages Locally

Initially make sure your local Linux machine installed with the python version is the same as the lambda python runtime version.

#Install Python3
apt install python3

#Verify Python & pip version
python -V
pip3 -V

Create the local directory structure as follows and navigate to the python directory.

mkdir -p lambda/python
cd lambda/python

Then install the required python packages inside the python directory by specifying the “-t .” option (If you didn’t specify the -t flag it would be installed in the default Python site-packages location). Once it is installed, remove the unnecessary files (dist info and caches) to reduce the file size.

pip3 install -t . pandas
pip3 install -t . pretty-html-table

rm -r *dist-info __pycache__

Finally, package the required files into a zip file before upload into the AWS Lambda layers.

apt install zip #Ignore this if you already have installed zip
cd ..
zip -r9 lambda-layer.zip python

Create the Lambda Layer

Navigate to the AWS Lambda console and from t the left sidebar, select the Layers and create a new layer.

I have already uploaded the created zip file to the S3 bucket and here I’m using the “Upload a file from Amazon S3” option because sometimes direct upload has size limitations.

Add the Layer to the Lambda Function

Once the upload is done, go to the lambda function and under the Layers section, Add the layer as a Custom Layer.

Test the Lambda Function

Now, it’s time to test the function again. If you did the above steps correctly, now the lambda function works as expected!