Java OOM in multi-process fitnesse testing environment

Question

we are using fitnesse for acceptance testing of a complex web-based application. full suite would take several hours to pass, so we're using multiple processes. the setup is as follows:

1. maven forks fitnesse server process
2. maven makes http calls to fitnesse server
3. fitnesse server forks a test runner for each call
4. http call returns, go back to 2.

the command line to start jvm in 3. is built inside fitnesse jar based on http call parameters. meaning you can not pass in an arbitrary jvm argument, only ones that are supported.

now, the problem we're facing is that sometimes the process started in 3 hangs. it does not look like a timeout issue, because we have seen them hanging for multiple hours.

these jvm processes do not expose anything via jmx, so we can't connect to them using jconsole or alike. based on amounts of memory each process consumes (1-1.5G) i strongly suspect that OOM happens somewhere in the runner process and prevents it from exiting normally. also, trying "kill -3" on the server process produces

Exception in thread "CommandRunner stdOut" java.lang.OutOfMemoryError: Java heap space  at java.util.Arrays.copyOf(Arrays.java:3332)

in what seems to be stdout from the started runner process, but i'm not sure.

the solution i'm currently thinging of is creating my copies of classes that control the command line so that every process is started with a randomized jmx port and a debug port, so if it hangs - can connect and investigate.

so, the question is - is there a better way to do it? am i missing something obvious here?

Answer 1

I don't believe FitNesse's core is thread safe. So instead of focussing your efforts on controlling how the runners are starting, and debugging those I recommend you try running your test-processes completely isolated from each other. By using a single process for FitNesse and the test runner (and having multiple of these when you want to run suites in parallel) it also becomes easier to debug, if there are any issues (as you don't have to control the creation of extra processes and attach to these new processes, there is only on process for a test run/suite).

What I do is I run my test suites via the jUnit runner (which uses a single process to run both FitNesse and the test runner) and start a separate (jUnit) process per test suite I want to run in parallel. I don't actually control the process creation via maven, but use multiple processes in a Jenkins job. Each one executes the same maven command, but with different system properties and one of the system properties controls which suite to execute.

Using such an approach might be too big of a change for your test setup. But I believe you could change your process a little bit and get similar results. You have not specified exactly how your maven pom is structured. But could you not just have steps 1-4 repeated for each suite, instead of only steps 2-4. This seems just to require changing the port that the FitNesse server listens on (in step 1) and that is easily done via command line parameters (-p). Running in the same process as the wiki should be possible by adding &debug to the URL used in step 2.