aiboto3 in Lambda Function Container: Runtime Error: No module named 'botocore.compress'

Question

I use containerized lambda functions.

This is the Dockerfile:

FROM public.ecr.aws/lambda/python:3.10

COPY requirements.txt .
RUN pip install -r requirements.txt
RUN pip install --upgrade boto3 # Just to be safe
RUN pip install --upgrade botocore # Just to be safe

COPY ./main.py ${LAMBDA_TASK_ROOT}

CMD ["main.lambda_handler"]

This is main.py.

import asyncio
import aioboto3


def lambda_handler(event, context):
    key = "test"
    bucket = "test"
    loop = asyncio.get_event_loop()
    result = loop.run_until_complete(test_key(bucket, key))
    return result


async def test_key(bucket: str, key: str):
    try:
        session = aioboto3.Session()
        async with session.client("s3") as client:
            response = await client.head_object(Bucket=bucket, Key=key)
            return True
    except Exception as e:
        return False

requirements.txt

aioboto3
asyncio

When invoking the function (either locally or inside an online Lambda environment) I get the following error:

{"errorMessage": "Unable to import module 'main': No module named 'botocore.compress'", "errorType": "Runtime.ImportModuleError", "requestId": "87dcc2be-d84c-4725-8f2f-663219c6f43c", "stackTrace": []}

Answer 1

It appears that this has to do with how AWS is running the Python images. From my experiments outlined below, it does not appear that Lambda uses the botocore inside the image; the image is treated as just a "box" for the files. The execution environment, including the version of botocore, is provided by the Lambda service.

At the time of writing, Python 3.9 and 3.10 images are using boto3+botocore 1.26.90, while Python 3.11 using 1.34.34. As Bert have highlighted above, botocore>=1.31.14 is required for botocore.compress, so currently Python 3.9 and 3.10 would not be able to run aioboto3 at all.

The test is conducted by a simple lambda_handler of:

import json
import sys


def lambda_handler(event, context) -> dict:
    import boto3
    import botocore
    try:
        import aioboto3

        return {
            "statusCode": 200,
            "body": json.dumps(
                {
                    "python": sys.version,
                    "aioboto3": aioboto3.__version__,
                    "boto3": boto3.__version__,
                    "botocore": botocore.__version__,
                }
            ),
            "headers": {"Content-Type": "application/json"},
        }

    except Exception as e:
        return {
            "statusCode": 500,
            "body": json.dumps(
                {
                    "python": sys.version,
                    "boto3": boto3.__version__,
                    "botocore": botocore.__version__,
                    "error": {"kind": type(e).__name__, "message": str(e)},
                }
            ),
            "headers": {"Content-Type": "application/json"},
        }

Dockerfile:

FROM public.ecr.aws/lambda/python:3.10 AS app
ARG PIP_INSTALL

RUN ${PIP_INSTALL} aioboto3

COPY image/ .

CMD ["app.lambda_handler"]

public.ecr.aws/lambda/python:3.11 returns:

{'python': '3.11.6 (main, Dec  4 2023, 13:34:04) [GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]',
 'aioboto3': '12.3.0',
 'boto3': '1.34.34',
 'botocore': '1.34.34'}

public.ecr.aws/lambda/python:3.10 returns:

{'python': '3.10.13 (main, Dec  4 2023, 13:30:46) [GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]',
 'boto3': '1.26.90',
 'botocore': '1.29.90',
 'error': {'kind': 'ModuleNotFoundError',
  'message': "No module named 'botocore.compress'"}}

public.ecr.aws/lambda/python:3.9 returns:

{'python': '3.9.18 (main, Dec  4 2023, 13:30:08) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]',
 'boto3': '1.26.90',
 'botocore': '1.29.90',
 'error': {'kind': 'ModuleNotFoundError',
  'message': "No module named 'botocore.compress'"}}


In a nutshell - try migrating your code to py3.11 if possible!