Invoking a Lambda function that has a Python library dependency in Docker container, deployed with Zappa; 500 response code error

Question

I am looking to invoke AWS Lambda functions, deployed using Zappa. The lambda functions will have dependencies that are too large for the usual zip file approach (even with slim_handler : true).

To do this I am using Zappa with Docker containers, where the container holds the larger dependencies and the Lambda functions call upon the container as needed.

The application is a Flask app, with the usual app.py code that routes to a function. Deploying a dockerized flask app with Zappa is pretty straightforward, and I can do this successfully, as long as there is only vanilla Python in the app.py. But as soon as I add Python that depends on a library, I get the following error when attempting to invoke the Lambda function via the AWS API Gateway (as setup by Zappa):

Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 500 response code.

Here are the steps I am taking. I am using Feast as the Python library I want to work, with the dockerized Flask app pushed to Lambda with Zappa.

I Create the Dockerfile that works alongside Zappa for deploying to AWS Lambda.

I am using the steps outlined here to create this Dockerfile:

FROM amazon/aws-lambda-python:3.8

ARG FUNCTION_DIR="/var/task/"

COPY ./ ${FUNCTION_DIR}

# Setup Python environment
RUN pip install pipenv

RUN pip install -r requirements.txt

# Grab the zappa handler.py and put it in the working directory
RUN ZAPPA_HANDLER_PATH=$( \
    python -c "from zappa import handler; print (handler.__file__)" \
    ) \
    && echo $ZAPPA_HANDLER_PATH \
    && cp $ZAPPA_HANDLER_PATH ${FUNCTION_DIR}

CMD ["handler.lambda_handler"]

Briefly, this uses a base image provided by AWS, copies my application code into the image, and sets up my Python environment with pipenv. The Zappa specific steps are at the bottom, where handler.py is manually added to the Docker image. "The lambda_handler function contains all the Zappa magic that routes API Gateway requests to the corresponding Flask function." The handler is specified in the CMD so that it runs whenever a Docker container using this image is started.

Here is my requirements.txt file:

zappa
flask
feast

Since I want Feast to be available, I install this inside the virtual environment:

Installing FEAST

These steps are from here:

Install Feast:

pip install feast

Create a Feature Repo:

feast init feature_repo
cd feature_repo

Register feature definitions and deployed feature store:

feast apply

Load features into my online store:

CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
feast materialize-incremental $CURRENT_TIME

To test that Feast will be successfully deployed on Lambda I will fetch the feature vectors for inference over the AWS API Gateway, by invoking a Lambda Function over AWS. So I need the necessary Feast code callable by Flask:

Fetching Feature Vectors for Inference

I set up the Feast code inside the usual Flask app.py:

from flask import Flask
    
from feast import FeatureStore

app = Flask(__name__)

@app.route('/')
def index():

    store = FeatureStore(repo_path="feature_repo/")

    feature_vector = store.get_online_features(
        feature_refs=[
            "driver_hourly_stats:conv_rate",
            "driver_hourly_stats:acc_rate",
            "driver_hourly_stats:avg_daily_trips",
        ],
        entity_rows=[{"driver_id": 1001}],
    ).to_dict()

    return feature_vector

# We only need this for local development.
if __name__ == '__main__':
    app.run()

The expected response from this is:

{
    'driver_id': [1001],
    'driver_hourly_stats__conv_rate': [0.49274],
    'driver_hourly_stats__acc_rate': [0.92743],
    'driver_hourly_stats__avg_daily_trips': [72],
}

I also install Zappa inside the virtual environment:

pipenv install zappa

...here is my zappa_settings.json:

"zappa_test": {
    "app_function": "app.app",
    "project_name": "app",
    "runtime": "python3.8",
    "s3_bucket": "zappa-f8s0d8fs0df",
    "aws_region" : "us-east-2",
    "lambda_description": "Zappa + Docker + Flask"
}

The S3 bucket number shown here is fake.

Also installing Flask inside the virtual environment:

pipenv install flask

Building the Image

I run the following to build the image:

zappa save-python-settings-file zappa_test
docker build -t zappa_test:latest .

I then Push to ECR:

aws ecr create-repository --repository-name zappa_test --image-scanning-configuration scanOnPush=true

Re-Tag the Image:

docker tag zappa_test:latest XXXXX.dkr.ecr.us-east-1.amazonaws.com/zappa_test:latest

Get authenticated to push to ECR:

aws ecr get-login-password | docker login --username AWS --password-stdin XXXXX.dkr.ecr.us-east-1.amazonaws.com

...and push the image:

docker push XXXXX.dkr.ecr.us-east-1.amazonaws.com/zappa_test:latest

Finally, I deploy the lambda function:

zappa deploy zappa_test -d XXXXX.dkr.ecr.us-east-1.amazonaws.com/zappa_test

So altogether, this is how one uses Zappa to push a docker image to AWS Lambda in order to invoke a lambda function over the AWS API Gateway. In this case, I am trying to invoke a lambda function that fetches Feast data via Flask, with the dependencies made available inside a Docker container.

But, upon running the deploy command I get:

Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 500 response code.

IMPORTANT:

I have confirmed that all of these steps work if I put something simple inside the app.py like this:

@app.route('/')
def index():
    return "Hello, world!", 200

Everything is credentialized on AWS correctly, the image deploys, and I can call it perfectly through the API endpoint provided by Zappa (via AWS Gateway). I only get the mentioned error when I put the Feast code inside app.py, as I've shown above.