OutOfMemoryError even though enough free memory

Question

I am getting java.lang.OutOfMemoryError errors, even when I still have enough free RAM. The memory dumps I took were between 200MB and 1GB, while my server has 24GB of RAM. I set -Xmx12288m -Xms12288m.

Also, when I try to log in to the server, I frequently get

-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

I narrowed it down to the code snippet below:

import org.snmp4j.Snmp;
import org.snmp4j.transport.DefaultUdpTransportMapping;

    long n = 0;
    while (true) {
        DefaultUdpTransportMapping transport = null;
        try {
            transport = new DefaultUdpTransportMapping();
            transport.listen();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
//      } finally {             // (*) I forgot this
//          transport.close();  // (*) I forgot this
        }

        n++;
        double freeMemMB = Runtime.getRuntime().freeMemory() / 1024 / 1024;
        System.out.println("Created " + n
                + " DefaultUdpTransportMappings. Free Mem (mb): "
                + freeMemMB);
    }

Output (on my developer machine, with mvn exec:java):

Created 2026 DefaultUdpTransportMappings. Free Mem (mb): 299.0
Created 2027 DefaultUdpTransportMappings. Free Mem (mb): 299.0
[WARNING] 
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at org.snmp4j.util.DefaultThreadFactory$WorkerThread.run(DefaultThreadFactory.java:91)
    at org.snmp4j.transport.DefaultUdpTransportMapping.listen(DefaultUdpTransportMapping.java:168)
    at App.main(App.java:19)
    ... 6 more

I found that I get the errors because I don't close the DefaultUdpTransportMapping. Enabling the finally { ... } block solves the problem. Now I'm wondering which limits (if not the amount of free memory) I reached. The ulimits on the server are:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 191968
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

On my developer Mac:

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                2560

Which limit did I reach?

Answer 1

The java.lang.OutOfMemoryError: unable to create new native thread is a confusing message, since it not really related to running out of heap memory. Therefore the heap size settings (Xmx and Xms) have no influence on this case.

The exception is thrown when a new operating process cannot be created for your application, either because a maximum number of processes/open file handles is reached or there is no memory left on the system to create a new thread.

Regarding the ulimit settings, it can be either the number of file descriptors, the stack size or the number of processes. The stack size is a per thread number. The number of threads times the stack size will be the amount of used memory.

Usually, as it is in your case, getting this exception means your application is not closing its threads properly and keeps hold of system processes. This is why closing the transport fixed the issue for you.