Chaos Monkey delay test causes tomcat-jdbc connection pool to have huge delay in getting connection

Question

We are running delay test using Chaos Monkey on our service. We introduce 2 second delay using it.

During this time the response time of our service goes down to 35-40 seconds also.

When we profiled our application, we found that the maximum amount is being spent on getting connection from connection pool. We use tomcat-jdbc for connection pooling.

Ideally the 2 second delay should have cause <5 second deterioration in our response time.

We checked the logs of tomcat-jdbc and found waitCount increasing.

We increased the InitialSize and MaxActive connections in connection pool. It helped a little but nothing changed significantly.

What can be possibly going wrong here?

Answer 1

As you don't really give metrics about the app behavior, it is difficult to provide an accurate answer but usually this is what happens when there is some network issue about the DB in a tomcat Java app :

• JDBC connection pool is exhausted quickly, all connections are in use by a thread.
• Threads trying to answer to requests wait for a connection from the pool for very long time
• More and more threads wait for a connection, so the wait time increases.

What you need to do is probably to decrease some timeout settings, for example the "maxWait" parameter.