Increase s3 connection pool

Question

I have the following code:

    client_config = botocore.config.Config(
        max_pool_connections=20
    )
    athena = boto3.client('athena')
    s3 = boto3.resource('s3',config=client_config)
    
    query_result = athena.start_query_execution(
        QueryString = query,
        ResultConfiguration = {
            'OutputLocation': s3_url
        }
    )
    queryExecutionId = query_result['QueryExecutionId']
    response = athena.get_query_execution(QueryExecutionId = queryExecutionId)
    pd.read_csv(f"s3_bucket/{queryExecutionId}.csv")
    
    athena.close() 

It throws the following warning:

Connection pool is full, discarding connection: x.s3.us-west-2.amazonaws.com. Connection pool size: 10

How can I increase this pool size for s3 connection?

Thanks.

Answer 1

Just a short information about S3

S3 did not work as a connection pool at all, it has a limit of 3000 connections a sec as limit for each portion, if you have a lot of files and you need speed you need to use folder structures as you portions eg.

/folder1/ = 3000 connection
/folder2/ = 3000 connection
/folder3/ = 3000 connection
/folder4/ = 3000 connection
/folder5/ = 3000 connection

It will give you up to 15.000 connections a sec, if you need more than 3000 in eg. folder1 then you need to again add a subfolder like this

/folder1/a/ = 3000 connection
/folder1/b/ = 3000 connection
/folder1/c/ = 3000 connection

Short info about Athene

If this is not a problem I can see you using Athena, and the limit for Athena queries is 25 runs at the same time, read more about the quotes on AWS here https://docs.aws.amazon.com/athena/latest/ug/service-limits.html

Awsner for you question

Based on Boto3 documentation you should be able to change the pool size by changing the max_pool_connections in the botocore.config configuration.

Read more here https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Hope its help on your problem at all :)