activemq memory usage 4 times greater than Maximum heap allocated

Question

I have below configuration for my java actvemq process where Xmx is 15360M but when I see the process usage of memory using glance I see RSS 41.8gb and VSS 51.6gb . pmap |grep total gives me total 54059348K. I am not sure what is happening in activemq that is causing this much amount of memory being utilized. While heap usage is pretty low till 60%.

In the output of pmap I see lot of anon blocks. Please find attached PMAP output

/XXXXX 111922      1 22 14:30 pts/0    00:04:26 
/u/users/user/jdk1.7.0_71/bin/java -Xms1024M -Xmx15360M -Xss512K 
-XX:+UseG1GC -XX:NewRatio=4 -XX:InitiatingHeapOccupancyPercent=45 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/x/logs/AMQGC.log 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=20M 
-Djava.util.logging.config.file=logging.properties 
-Djava.security.auth.login.config=/u/users/user/POSInterfaces/apache-activemq-5.13.2//conf/login.config 
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote 
-Djava.awt.headless=true 
-Djava.io.tmpdir=/u/users/user/POSInterfaces/apache-activemq-5.13.2//tmp 
-Dactivemq.classpath=/u/users/user/POSInterfaces/apache-activemq-5.13.2//conf:/u/users/user/POSInterfaces/apache-activemq-5.13.2//../lib/: 
-Dactivemq.home=/u/users/user/POSInterfaces/apache-activemq-5.13.2/ 
-Dactivemq.base=/u/users/user/POSInterfaces/apache-activemq-5.13.2/ 
-Dactivemq.conf=/u/users/user/POSInterfaces/apache-activemq-5.13.2//conf 
-Dactivemq.data=/u/users/user/POSInterfaces/apache-activemq-5.13.2//data 
-jar /u/users/user/POSInterfaces/apache-activemq-5.13.2//bin/activemq.jar 
start/ 

Please find pmap file here

Answer 1

I can confirm that ActiveMQ is unable to use any more JVM heap than you configure with -Xmx. ActiveMQ is 100% Java, so no other heap memory than that of the JVM will be used by the broker.

To find out how much Java heap is really used, query the JMX statistics of the JVM, it should always remain within the configured limit.

I am not familiar with pmap, so I cannot really comment on that part but am wondering what numbers the top command gives you?

Answer 2

I take it from reference to RSS that you are running on Unix - not sure if that's Linux.

I'm wondering if you should be looking at PSS (Proportional Set Size) rather than RSS (Resident Set Size) - see this Unix & Linux Question on memory - I have found smem to be a great tool for assessing actual memory usage.

For example - for an ActiveMQ process on my server (set with -Xmx512m)

[user@vm ~]$ sudo pmap 13727 | grep total
 total          3161064K

[user@vm ~]$ sudo smem | grep -e '13727' -e 'PSS'
  PID User     Command                         Swap      USS      PSS      RSS
13727 activemq /usr/bin/java -Xms64M -Xmx1        0   212604   214962   229468

i.e. pmap giving me a much higher memory figure (3GB) than smem PSS (and indeed RSS) (~200M)

Answer 3

To profile memory usage of your application (including not only Java Heap), you can use this answer.

Most of your anon blocks have close size. In most cases a lot of anon talk about threads. You could have a lot of threads, if you have a lot of destinations in ActiveMQ. But sizes of your anon blocks are not close to Thread Stack Size. So it could be not threads, I suppose.

Note that 15Gb is very high for ActiveMQ and sometimes for Java too, it could affect GC very strong. You can read how to configure ActiveMQ for system memory usage here and here.