Home Cloud Import Python Libraries for AWS Lambda Layers

Import Python Libraries for AWS Lambda Layers

by Lakindu Jayasena
8.4k views 8 mins read
Import Python Libraries for AWS Lambda Layers

Are you getting libraries or dependencies missing errors when you work with your lambda function code with python? Do you want to keep your lambda deployment package smaller and cleaner? Here you are in the right place…

Overview of AWS Lambda Layers

AWS Lambda Layer is simply archived files that contain additional libraries or other dependencies that you can use at runtime for your lambda workloads. That is very useful if you have multiple Lambda functions that use the same set of libraries or dependencies and improve the reusability of packages.

When you add a lambda layer into a lambda function, the contents of the archived files are extracted to the /opt directory in the execution environment. You can have up to five layers per function and it increases the lambda deployment size limits. The lambda layers are deployed as a fixed version and the version number gets increment each time when you add a new layer. When you add a layer to a lambda function, you have to specify the layer version and that will help you to backward compatibility with older versions of dependencies.

Benefits of AWS Lambda Layers

  • Make your workload into modular and easily deploy.
  • By default lambda layers enabled with versioning and allow you to use previous versions when needed.
  • Since the layer can be use across multiple lambda function package re-usability is there.
  • Very easy to do the changes quickly via the AWS lambda console.

Let’s get started!

In this post, I’m going to explain how to locally build and import external python libraries (pandas & pretty_html_table) and add it to lambda layers in your own lambda workload.

Unable to import module ‘lambda_function’?

In a normal way, if you try to import pandas python library directly into your lambda code, you will get an error as follows.

Sample Python Code on AWS Lambda
AWS Lambda Python Packages Error

The problem here is there is no way to install python modules like you are using in your local machine using pip commands. Ok, then the next question is how can I add these libraries to the AWS Lambda?

Here is the Solution

Prepare the Required Packages Locally

Initially make sure your local Linux machine installed with the python version is the same as the lambda python runtime version.

 #Install Python3 apt install python3 #Verify Python & pip version python -V pip3 -V 

Create the local directory structure as follows and navigate to the python directory.

 mkdir -p lambda/python cd lambda/python 

Then install required python packages inside the python directory by specifying the “-t .” option (If you didn’t specify the -t flag it would be installed in the default Python site-packages location). Once it is installed, remove the unnecessary files (dist info and caches) to reduce the file size.

 pip3 install -t . pandas pip3 install -t . pretty-html-table rm -r *dist-info __pycache__ 
Python Packages Installation
Note: You can use macOS/Windows machine to achive the same, but that would be more complicated because some packages having different precompiled binaries for specific OS architecture and it would not be compatible with AWS Lambda python runtime.

If you really need to do it on macOS/Windows environment, you need to download the wheel files (.whl) for linux and extract it inside directory and manually add the wheel file package dependacies too.
So you can eliminate all the hassles if you do this on linux envitonment with required python version.

Finally, package the required files into a zip file before upload into the AWS Lambda layers.

 apt install zip #Ignore this if you already have installed zip cd .. zip -r9 lambda-layer.zip python 

Create the Lambda Layer

Navigate to the AWS Lambda console and from t the left sidebar, select the Layers and create a new layer.

Navigate to Lambda Layers

I have already uploaded the created zip file to the S3 bucket and here I’m using the “Upload a file from Amazon S3” option because sometimes in direct upload having size limitations.

Upload the created package via S3

Add the Layer to the Lambda Function

Once the upload is done, go to the lambda function and under the Layers section, Add the layer as a Custom Layer.

Add the AWS Lambda Layer to Lambda function

Test the Lambda Function

Now, it’s time to test the function again. If you did the above steps correctly, now the lambda function works as expected!

AWS Lambda Function Test Sucess

Related Articles

3 comments

Anonymous April 19, 2022 - 2:43 PM

Thanks, this helped.

Reply
Shasini July 18, 2022 - 12:36 PM

Tried this. but it still gives the

{
“errorMessage”: “Unable to import module ‘Test’: No module named ‘datadog_api_client'”,
“errorType”: “Runtime.ImportModuleError”,
“stackTrace”: []
}

Reply
Admin Avatar
Lakindu Jayasena July 18, 2022 - 5:49 PM

Check the lambda function python version and locally install python versions (to prepare the packages locally) are the same.

Reply

Leave a Comment

* By using this form you agree with the storage and handling of your data by this website.