Running Tomcat v9.0.80 using Open JDK (JRE Only) v16.0.2 on a Windows Server.
openjdk version "16.0.2" 2021-07-20
OpenJDK Runtime Environment (build 16.0.2+7-67)
OpenJDK 64-Bit Server VM (build 16.0.2+7-67, mixed mode, sharing)
It is crashing multiple times a day. The Tomcat process stays up but Tomcat is unresponsive. It is run as a Windows Service. This occurs multiple times a day and necessitates restarting Tomcat until it occurs again. From what I can see, it only occurs when there's a load. If there's little or no activity, Tomcat responds well.
From the Tomcat Management Console, I see that Memory is normal. There are no out-of-memory exceptions in the Tomcat logs. No DB connection leaks. In my experience, DB connection leaks (failing to close a connection) can lead to memory leaks.
What I do see in the Tomcat Management Console is the Current Thread Count climbs dramatically in a few minutes, after the Current threads Busy Number climbs dramatically. Then it eventually reaches the Max Threads count and you get a 503 Service Unavailable and Tomcat has to be restarted. It's every single thread. Stage is all S (Service).
So, let's say the Max Threads is 200, Current Thread Count is 100, and Current Threads Busy is 0. In a matter of minutes, Current Threads Busy sharply climbs from 0 to 100. Then Current Thread Count goes to 101 and Current Threads Busy goes to 101. The two are in-sync, as the threads are busy and the web server is hanging but not HTTP 503 just yet. Both Current Threads Count and Current Threads Busy eventually reach the threshold of 200 and then the web server reports HTT 503 Service Unavailable.
This happens multiple times a day but only with a load. Not sure what is causing it yet. I think the Threads is a symptom of the real problem(s), whatever that may be.
Any ideas on what this could be and how to solve it?