JPype / Java - Initialize with, or get, remaining heap space

Question

We have software written in Python, which uses JPype to call Java, which performs various resource heavy calculations / report building. We originally assigned 800mb of heap space when starting the JVM. The java side is fully multithreaded and will work with whatever resources are available to it.

jvmArgs = ["-Djava.class.path=" + classpath, "-Xmx800M"]
jpype.startJVM(u"java\\jre8\\bin\\client\\jvm.dll", *jvmArgs)

This worked well until we tested on Windows XP for our legacy clients. The new machines are Win 7 64-bit with 4GB of RAM, whereas the old ones are Win XP 32-bit with only 2 GB of ram.

The issue is that JPype causes our application to ungracefully and silently crash if we allocate too much memory. A try catch doesn't even get triggered on the statement above.

I'm wondering if there's a way to use java from command line to determine how much memory we can allocate on a computer. We can check if it's 32-bit or 64-bit which helps, but we need to make sure they aren't running other programs taking up heap space on the JVM. If they are, our application will crash.

Reader's Digest: We'd like to allocate 500mb of heap space when initializing the JVM, but can't be sure of how much space is currently being used. If we allocate too much, the entire application silently crashes.

We use the following JPype: 0.5.4.2 Python: 2.7 Java: 1.8 or 1.7 (64-bit or 32-bit)

Thanks.

Answer 1

The memory consumed by JVM consists of 2 main areas:

1. Heap memory
2. Non heap memory - MetaSpace, native method stacks, pc register, direct byte buffers, sockets, jni allocated memory, thread stacks and more

While the maximum size that will be used for the heap memory is known and configurable, the size of the non heap memory cannot be fully controlled.
The size of the native memory used by the JVM will be effected by the number of threads you use, the amount of classes being loaded and the use of buffers (use of I/O).
You can limit the size of the metapsace by setting the MaxMetaspaceSize (-XX:MaxMetaspaceSize). You can control the amount of memory used for thread stacks by limiting the number of threads and setting the thread stack size (-Xss). Assuming you do not have native memory leaks, the amount of classes being loaded is stable (no excessive use of dynamic proxies and bytecode generation) and the amount of threads being used is known - you can speculate how much memory will be required for your application to run by monitoring the overall memory used by the JVM over a period of time. When you do it, make sure the entire heap is being allocated when the JVM starts.