What are some common causes of Java Direct buffer memory errors?

Question

I have a variety of Rails applications running on Torquebox. Occasionally, Torquebox will stop responding to requests for a short period of time (maybe 2-5 minutes) and the logs will fill up with the following error messages:

java.lang.OutOfMemoryError: Direct buffer memory

The errors happen at unpredictable times (often days between). Load testing doesn't reproduce the problem, and the issue doesn't happen during peak load anyway. And, unlike many other types of memory errors I've seen in the past, the server actually recovers and starts responding normally again without any sort of restarts or intervention.

Are there any common coding mistakes, misconfigurations, or other potential problems that regularly cause this error? Google reveals lots lower level Java/Garbage collection type issues with various libraries (Netty for example), but I'm interested to see if there are other common places to look as well.

Answer 1

JNA/ByteBuffer not getting freed and causing C heap to run out of memory indicates that the direct memory might not be cleared if the Java heap doesn't require garbage collection very often (perhaps, in your case at non-peak times).

If you have a constant function using the direct memory, regardless of load, then the application might not be calling the garbage collection enough during the lighter load times. Using the GC module might help (http://ruby-doc.org/core-2.0/GC.html).

Answer 2

The problem in this case ended up being lack of enough total available memory. Java's memory footprint can end up being significantly larger than just the heap allocation. Direct Buffer Memory is only one of the types of memory use that falls outside the heap.

It's still not clear to me why the fluctuations occur at unpredictable times, but ensuring there is enough excess memory on the system to handle some fluctuation is critical to stability.