I have a very big ulimit -n 300k for the user.
My Java program encounter “Too many open files” exception. To debug this, I run lsof -p PID | wc -l and the output value is a lot less than that ulimit. That means I don't reach FD limit.
I use jmx getOpenFileDescriptorCount inside the application to print periodically, but it shows usage as high as ulimit number and when it reach that Java throw exception. In the mean time, lsof -p PID is only 1/60 of ulimit, which make sense for my application which only has certain number of TLS network connections.
I’m running openjdk1.8 in Centos, what’s the problem? How to debug?