Database connections count greater than that set by DBCP maxTotal

Question

I am running a java web application using Tomcat (version 8.0.43).

I moved the tomcat-dbcp.jar (which came with Tomcat8.0.43) to jdk/jre/lib/ext so that it's available to the jre.

I think that I am using the Tomcat DBCP (as opposed to Apache Commons DBCP) but as I understand from the documentation, the configuration parameters are the same as for Apache Commons DBCP, which are detailed here.

Accordingly, I set up the following Resource element for the connection pool in context.xml:

<Resource 
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
auth="Container" 
driverClassName="com.mysql.jdbc.Driver"  
logAbandoned="false" 
maxTotal="100" 
maxIdle="30" 
maxWaitMillis="10000" 
name="jdbc/[dbName]" 
password="${dbPassword}" 
removeAbandonedOnBorrow="true" 
removeAbandonedTimeout="60" 
type="javax.sql.DataSource" 
url="[dbUrl]autoReconnect=true" 
username="${dbUsername}" 
testWhileIdle="true" 
testOnBorrow="true" 
validationQuery="SELECT 1 AS dbcp_connection_test"/>

Lately, I have been getting an error in my logs:

MySQLNonTransientConnectionException: Too many connections

So I checked how many connections I have at the time of the error (via Amazon CloudWatch for RDS. Specifically "ConnectionCount") and it can reach as high as 150 connections. How is this possible if I have set my maxTotal connections to 100?

I am running my application on multiple instances. Can it be that the maxTotal attribute applies to each instance (eg: if I have the Connection Pool defined on 2 instances then my maxTotal = 100 + 100 = 200?)

It's as if the attributes that I set in the Resource element are ignored. Can it be that something is wrong with my setup? Am I not using the tomcat DBCP like I assumed?

In addition, if I look further down the StackTrack of the Exception that I quoted above, I see that

org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection

is called. I'm not sure if this means that I am using a different Connection Pool (DBCP2) from the one I assumed I'm using (Tomcat DBCP).

Answer 1

Can it be that the maxTotal attribute applies to each instance (eg: if I have the Connection Pool defined on 2 instances then my maxTotal = 100 + 100 = 200?)

That's correct since you have defined your <Resource> inside the context.xml file.
Add your resource inside the server.xml to make it available to all contexts and make sure you remove it from all context.xml files. Make note of the global attribute not present on your configuration

<GlobalNamingResources>
    <Resource
        global="jdbc/[dbName]"

        factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
        auth="Container" 
        driverClassName="com.mysql.jdbc.Driver"  
        logAbandoned="false" 
        maxTotal="100" 
        maxIdle="30" 
        maxWaitMillis="10000" 
        name="jdbc/[dbName]" 
        password="${dbPassword}" 
        removeAbandonedOnBorrow="true" 
        removeAbandonedTimeout="60" 
        type="javax.sql.DataSource" 
        url="[dbUrl]autoReconnect=true" 
        username="${dbUsername}" 
        testWhileIdle="true" 
        testOnBorrow="true" 
        validationQuery="SELECT 1 AS dbcp_connection_test"/>
</GlobalNamingResources>

To keep an eye on:

• testOnBorrow="true" causes a performance penalty that is not always worth it.

• testWhileIdle="true" is related to timeBetweenEvictionRunsMillis which is 5 seconds by default. Default value could be a too short period in some cases.

• Verify AWS RDS quota is not exceeded.