Is it possible to correctly point to Python Shapely library's LIBGEOS_C in AWS Lambda environment?

Question

I am attempting to write a AWS python Lambda function that utilize's Shapely for simple "point in polygon" operations. I spun up an AWS linux EC2 instance, installed Shapely and had a working script. I then downloaded the linux specific libgeos_c.so.1 binary from my EC2 instance and bundled it (via serverless framework) to exist in the same directory as my lambda function. However once I deploy the script will not execute because it throws a "Could not find library or load any of its variants..." error. I've even tried to explicitly point to libgeos_c.so.1 path via the GEOS_LIBRARY_PATH environment variable to no effect. Is this an impossible deployment?

Here is a code snippet that sets the environment variable and then invokes a secondary script which actually imports and utilizes shapely.

import sys
import os
import subprocess

here = os.path.dirname(os.path.realpath(__file__))

# Import installed packages (in site-packages)
site_pkgs = os.path.join(here, "venv", "lib", "python2.7", "site-packages") 
sys.path.append(site_pkgs)
import json

def hello(event, context):

    command = "GEOS_LIBRARY_PATH={} python test_geo_worker.py".format(here + "/libgeos_c.so.1")
    foo = subprocess.check_output(command, shell=True)
    print foo

Has anyone every successfully deployed shapely in lambda? My fallback plan is to go back to good old postgres/postgis instead of shapely, but I'd sure like to try to build this out in a dynamo/lambda stack.

Answer 1

It is possible. Try to follow these steps and see if the problem persists.

First, install GEOS:

yum install geos-devel.x86_64

Then Shapely:

pip2.7 install shapely

Try to import Shapely:

python2.7 -c "import shapely"

Answer 2

As I have hit this problem and managed to find a solution, I thought I'd post it here.

Problem is that he necisary compiled libraries don't exist within Lambda, if you include libraries compiled by some other Linux, unless they are built with the same compiler and dependencies are not going to work.

Thankfully a nice chap has figured out the dependencies and built packages for a variety of Python modules that are not included in Lambda, including shapely.

https://github.com/ryfeus/lambda-packs

download the relevant module from there and copy it into the deployment package (removing any you have installed via pip beforehand).

Answer 3

I managed to get it running using images from https://hub.docker.com/r/lambci/lambda/tags

Make sure to grab the image with the right tag: e.g. python3.7

You can either just run a container off of that directly and compile your libraries and push in that environment, or you can follow the instructions here to build your own docker image and automate installing your packages, creating a ZIP with your code, and uploading the ZIP to lambda/s3.

Answer 4

I setup a build script that will produce the Shapely dependencies as a Lambda Layer. You can check out my project here https://github.com/bearflagrobotics/libgeos-lambda-build but basically it's just using an Amazon image to download and build Shapely's C++ dependencies for use in Lambda. Just drop in the zip file as a layer and run any version of Python/Shapely