Create Python Lambda Layer using Docker

AWS Lambda now supports bringing our own runtime environment using the Lambda layer. It’s a .zip file archive that can contain additional code or data, libraries, a custom runtime, or configuration files. It helps to share and separation of responsibilities so that you can iterate faster on writing business logic.

Lambda Layer Limitation

If your zip file is less than 50MB, you can directly upload it into the lambda layer. Otherwise, you have to upload your package to S3 and then deploy it to the lambda service.

A lambda function can use up to 5 layers now. Max size of unzipped lambda and all layers can go up to 250MB.

When comes to Python Lambda, It’s bit tides to set the package folder into a lambda runtime environment. Especially if you use tried party python library in your project.

The following steps will help you to build all custom python packages into a folder, zip it and upload them to the lambda service as a lambda layer.

Steps to build Custom Python Lambda Layer

1. You need to install docker on your computer. You can download it from here
   
2. You need to have all your third-party python library in pip requirements file (requirements.txt)
   
   you can read this article about how to generate pip requirements documents from your project
   
   https://pip.pypa.io/en/latest/user_guide/#requirements-files
   
3. Base on your runtime version, you need to have the following folder structure with requirements.txt
   
   ├── requirements.txt
   └── python/
   └── lib/
   ├── python3.6/
   │ └── site-packages/
   └── python3.8/
   └── site-packages/
   Note: If you are using python 3.6 in lambda, you need only python->lib->python3.6->site-packages folder structure.
   
4. Now open up the command prompt or terminal, execute the following command. This command will use build-python3.6 as a base image and pip will read packages mention in requirements.txt, install all of them into the given folder.

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"

Note: Replace 3.6 with 3.7 or 3.8, depending on the compatible libraries that you want to install. If you notice any build error, execute the above command until all packages install in site-packages folder.

5. Now you have all libraries install in relevant folder under python runtime. You can execute following command to make this folder into zip and create layer package.

zip -r customLambdaLayer.zip python > .

6. Execute the following command to upload the zip file to AWS and create a lambda layer

aws lambda publish-layer-version --layer-name customLambdaLayer --description "My Custom Lambda Layer" --zip-file fileb://customLambdaLayer.zip --compatible-runtimes "python3.6" "python3.8"

Note: Remember to replace the zip file name with your package name and proper description.

7. You can now execute the following command or in the AWS console, you can configure the newly created custom lambda layer.

aws lambda update-function-configuration --layers arn:aws:lambda:<<Region>>:<<ACCOUNT NUMBER>>:layer:customLambdaLayer:1 --function-name <<YOUR FUNCTION NAME>>

Note: Remember to replace with right ARN for newly created lambda custom later.

Now all your custom package will be available in your lambda !

You can use SAM CLI to deploy custom lambda layer, read this article for more information

AWS SAM CLI