JMeter fails to create new threads because of OutOfMemoryError when testing with 10k+ users

Question

My test makes one POST request then makes N PUT requests where N is the number of users. I also save the responses inside target

When testing with 10,000+ users JMeter fails with:

Uncaught Exception java.lang.OutOfMemoryError: Java heap space in thread Thread[MyCoolTest 1-3165,5,main]. See log file for details.

It looks like I am running out of memory after awhile.

1. What is the root cause?
2. Is there a way to increase memory for JMeter when I run it inside CircleCI's docker images? (I assume I can locate the jmeter.bat file and increased Xmx.

Some article I found related to this issue is here

Enviroment: JMeter runs in CircleCI linux machines:

• Operating System: Ubuntu 20.04.5 LTS
• OSType: linux
• Architecture: x86_64
• Java version: openjdk:11.0
• JMeter version: 5.5

Dependency Mngmt: Maven installs JMeter inside docker image

More details from logs:

java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61) ~[?:?]
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:348) ~[?:?]
    at org.apache.logging.log4j.core.layout.StringBuilderEncoder.getThreadLocalState(StringBuilderEncoder.java:82) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:64) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:231) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.core.Logger.log(Logger.java:161) ~[log4j-core-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2198) ~[log4j-api-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2152) ~[log4j-api-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2135) ~[log4j-api-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2028) ~[log4j-api-2.13.1.jar:2.13.1]
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1899) ~[log4j-api-2.13.1.jar:2.13.1]
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:184) ~[log4j-slf4j-impl-2.13.1.jar:2.13.1]
    at org.apache.jmeter.threads.JMeterThread.initRun(JMeterThread.java:712) ~[ApacheJMeter_core-5.3.jar:5.3]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:252) ~[ApacheJMeter_core-5.3.jar:5.3]
    at java.lang.Thread.run(Thread.java:829) [?:?]

Don't know where to start

Answer 1

1 HEAP is the allocated memory during startup of the JVM on your RAM memory. So the goal of 10,000 VU depends on the physical limitations of the machine you run it on. If you want to take the maximum, BlazeMeter suggests:

"Fortunately, there’s a simple solution. Just increase the maximum heap size to ~80% of your total available physical RAM. To do this, find the following line in your JMeter startup script:"

Inside jmeter.bat you can change the HEAP by setting the correct -Xms[MEMORY_G] and -Xmx[MEMORY_G]:

Example for a 16GB RAM machine would be -Xms12g and -Xmx12g.

set HEAP=-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m

2 Docker documents: "By default, a container has no resource constraints and can use as much of a given resource as the host’s kernel scheduler allows."

So if you run the Docker with default options, it eats up all RAM used by the container.

Answer 2

If you are starting the JMeter process with the JMeter maven plugin you can specify the Xms and Xmx in your POM file under plugin configuration:

<project>
[...]
    <build>
        <plugins>
            <plugin>
                <groupId>com.lazerycode.jmeter</groupId>
                <artifactId>jmeter-maven-plugin</artifactId>
                <version>3.8.0</version>
                <executions>
                    <!-- Generate JMeter configuration -->
                    <execution>
                        <id>configuration</id>
                        <goals>
                            <goal>configure</goal>
                        </goals>
                    </execution>
                    <!-- Run JMeter tests -->
                    <execution>
                        <id>jmeter-tests</id>
                        <goals>
                            <goal>jmeter</goal>
                        </goals>
                    </execution>
                    <!-- Fail build on errors in test -->
                    <execution>
                        <id>jmeter-check-results</id>
                        <goals>
                            <goal>results</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <jMeterProcessJVMSettings>
                        <xms>1024</xms>
                        <xmx>1024</xmx>
                        <arguments>
                            <argument>-Xprof</argument>
                            <argument>-Xfuture</argument>
                        </arguments>
                    </jMeterProcessJVMSettings>
                </configuration>
            </plugin>
        </plugins>
    </build>
[...]

This is on the JMeter Maven Plugin wiki at: https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/wiki/Configuring-the-jvm-that-the-jmeter-process-runs-in