I have built a deployment package with pandas, numpy, etc for my sample code to run. The size is some 46 MB. Doubt is, do I have to zip my code update every time and again update the entire deployment package to AWS S3 for a simple code update too?

Is there any other way, by which, I can avoid the 45 MB upload cost of S3 everytime and just upload the few KBs of code?

2 Answers

nosnahoj-trap On Best Solutions

I would recommend creating a layer in AWS lambda.

First you need to create an instance of Amazon Linux (using the AMI specified in https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html - at this time (26th of March 2019) it is amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2 ) or a docker container with the same environment as the lambda execution environment.

I personally do it with docker.

For example, to create a layer for Python 3.6, I would run

sudo docker run --rm -it -v "$PWD":/var/task lambci/lambda:build-python3.6 bash

Then I would create a folder python/lib/python3.6/site-packages in /var/task in the docker container (so it will be accessible later on in the directory on the host machine where I started docker)

do pip3 install <your packages here> -t python/lib/python3.6/site-packages

zip up the folder python and upload it as a layer and use it in my AWS lambda function.

NB! The paths in the zip file should look like "python/lib/python3.6/site-packages/{your package names}"

Now the heavy dependencies are in separate layer and you don't have re-upload them every time you update the function, you only need to update the code

Joseph On

Split the application into two parts. The first part would be the lambda function that only includes your application code. The next part a lambda layer. The lambda layer will include onky the the dependencies and be uploaded once.

A lambda layer can be uploaded and attached to the lambda function. When your function is invoked, AWS will combine the lambda function with the lambda layer then execute the entire package.

When updating your code, you will only need to update the lambda function. Since the package is much smaller you can edit it using the web editor, or you can zip it and upload it directly to lambda using the cli tools.

Exmample: aws lambda update-function-code --function-name Yourfunctionname --zip-file fileb://Lambda_package.zip

Here are video instructions and examples on creating a lambda layer for dependencies.It demonstrates using the pymsql library, but you can install any of your libraries there.