Too many open files Error in Cloud Foundry

Question

I have a java (tomcat / spring based) cloud multi-tenant application running on cloud foundry, which uses a cloud database. We use tomcat jdbc for connection pooling with maximum 8 connections per tenant. We have max wait configured for 60 seconds on connections in tomcat jdbc. We cache connection for 300 tenants at any given time.

Recently we experienced Too many open files Error.

When we checked lsof, we had some interesting findings:

1. There were 191 entries for each jar in lsof output. Does it mean, the jar file is being opened for each request?
2. There were many socket connections (to Cloud Database and Load Balancer - We have one http call to another micro-service in some calls) in CLOSE_WAIT state.

We further checked our dynatrace and found following:

1. We had some getConnection calls which failed with Broken Pipe error during borrowConnection (reconnect or validate), after 10 mins. I am not sure why max wait was not honoured, and is the socket closed after the Broken Pipe.
2. Eventually getConnection and jar loading (Don't know, why jar is being loaded again) started failing with Too many open files Error.

We also don't see any connection leak as it would resulted in PoolExhaustedException not the current one.

Any ideas / hints on what is going wrong?