I have an architecture where I use an Apache 2.2 to load balance 4 Tomcat 6.0.26 servers. All 4 servers are exactly same liferay instances and were running properly until a last couple of days. All my tomcat servers suddenly seem to stop and continuously throw the following exception -
Jun 22, 2015 4:54:42 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Jun 22, 2015 4:54:42 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Jun 22, 2015 4:54:42 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Jun 22, 2015 4:54:42 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Jun 22, 2015 10:46:26 AM org.apache.jk.common.ChannelSocket acceptConnections
WARNING: Exception executing accept
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
at java.net.ServerSocket.implAccept(ServerSocket.java:470)
at java.net.ServerSocket.accept(ServerSocket.java:438)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:312)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:666)
at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:877)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:679)
The "java.net.SocketException: Too many open files" exception is continuously thrown infinite times until the server memory is full and threshold reaches 100%. The only solution I found was to restart the servers and after every 12-14 hours same thing happens.