Are you getting libraries or dependencies missing errors when you work with your lambda function code with python? Do you want to keep your lambda deployment package smaller and cleaner? Here you are in the right place…
Overview of AWS Lambda Layers
AWS Lambda Layer is simply archived files that contain additional libraries or other dependencies that you can use at runtime for your lambda workloads. That is very useful if you have multiple Lambda functions that use the same set of libraries or dependencies and improve the reusability of packages.
When you add a lambda layer into a lambda function, the contents of the archived files are extracted to the /opt directory in the execution environment. You can have up to five layers per function and it increases the lambda deployment size limits. The lambda layers are deployed as a fixed version and the version number gets increments each time when you add a new layer. When you add a layer to a lambda function, you have to specify the layer version and that will help you to backward compatibility with older versions of dependencies.
Benefits of AWS Lambda Layers
- Make your workload modular and easily deploy.
- By default, lambda layers are enabled with versioning and allow you to use previous versions when needed.
- Since the layer can be used across multiple lambda function packages re-usability is there.
- Very easy to do the changes quickly via the AWS lambda console.
Let’s get started!
In this post, I’m going to explain how to locally build and import external python libraries (
pretty_html_table) and add it to lambda layers in your own lambda workload.
Unable to import module ‘lambda_function’?
In a normal way, if you try to import pandas python library directly into your lambda code, you will get an error as follows.
The problem here is there is no way to install python modules like you are using in your local machine using pip commands. Ok, then the next question is how can I add these libraries to the AWS Lambda?
Here is the Solution
Prepare the Required Packages Locally
Initially make sure your local Linux machine installed with the python version is the same as the lambda python runtime version.
#Install Python3 apt install python3 #Verify Python & pip version python -V pip3 -V
Create the local directory structure as follows and navigate to the python directory.
mkdir -p lambda/python cd lambda/python
Then install the required python packages inside the python directory by specifying the “
-t .” option (If you didn’t specify the -t flag it would be installed in the default Python site-packages location). Once it is installed, remove the unnecessary files (dist info and caches) to reduce the file size.
pip3 install -t . pandas pip3 install -t . pretty-html-table rm -r *dist-info __pycache__
If you really need to do it on macOS/Windows environment, you need to download the wheel files (.whl) for linux and extract it inside directory and manually add the wheel file package dependacies too.
So you can eliminate all the hassles if you do this on linux envitonment with required python version.
Finally, package the required files into a zip file before upload into the AWS Lambda layers.
apt install zip #Ignore this if you already have installed zip cd .. zip -r9 lambda-layer.zip python
Create the Lambda Layer
Navigate to the AWS Lambda console and from t the left sidebar, select the Layers and create a new layer.
I have already uploaded the created zip file to the S3 bucket and here I’m using the “Upload a file from Amazon S3” option because sometimes direct upload has size limitations.
Add the Layer to the Lambda Function
Once the upload is done, go to the lambda function and under the Layers section, Add the layer as a Custom Layer.
Test the Lambda Function
Now, it’s time to test the function again. If you did the above steps correctly, now the lambda function works as expected!
Thanks, this helped.
Tried this. but it still gives the
“errorMessage”: “Unable to import module ‘Test’: No module named ‘datadog_api_client'”,
Check the lambda function python version and locally install python versions (to prepare the packages locally) are the same.