tomcat dbcp _closed PoolableConnection but in ALLOCATED state

Question

The java application using ojdbc6.jar, tomcat 7, tomcat-dbcp-8.0.3.jar (and other jars that's probably not relevant to this question), JDK 7 (u51)

We have identified that there is a connection leak using v$session report where some connection gets into INACTIVE state for 7+ hours. This is also confirmed by Thread dump taken at frozen state.

Heap Dump (taken at frozen state) shows:

1. Total PoolableConnection and DefaultPooledObject is equal to maxTotal (expected for exhausted pool)
2. Each connection is associated to PooledObjectState ALLOCATED (expected)
3. lastReturnTime = lastUserTime = lastBorrowedTime (in DefaultPooledObject), which to me means: THREAD-1 (good workflow) returned the connection which was immediately borrowed by THREAD-2 (bad workflow having leak) and THREAD-2 never closed the connection, let it dangling!

ALL of the above observation makes sense, since we definitely have connection leak and eventual exhausted pool

My question is: When I see, details about PoolableConnection, it has associated boolean _closed which is "true". Why/How could it have "_closed = true". when I decompiled tomcat-dbcp jar, I could see that every time _closed is marked true, it will also associate IDLE state to connection object (instead of ALLOCATED).

Looking for theories on why this boolean is true.

PS: We have various ideas (like setting logAbandoned) to find the exact piece of code responsible for connection leak, I am looking forward to find a reason (or theory) for heap dump to capture these PoolableConnection _closed=true case.

Answer 1

Looking at DelegatingConnection source code it can be seen that closed=true can be set as the result of connection.close() or in a finally block after some exceptions as a safety measure.

} finally {
    closed = true;
}

There's a leak, the connection is in an inconsistent state because it could not be closed and is probably ready to be processed in the ABANDONED life cycle phase.
Inspecting the pool through JMX could give another perspective.
The leak could be related to an improperly handled exception that otherwise would have given a hint on the pool bad state.