JUnit Tests: Why is Maven (Surefire) so much slower than running on Eclipse?

Question

I'm working on a fairly big application, with ~260k LOC and roughly 1800 unit tests on the suite. A little background:

1. Multi-modular Maven project
2. All tests run on a separate module and JaCoCo is used to check coverage.
3. Tests are put together in some suites and then the suites are included on Surefire's configuration.
4. Using SpringJUnit4ClassRunner to run the tests
5. Surefire is currently configured to use the same VM to build and test (forkCountequals one and reuseForks equals true). Doing this I am sure I'm using the advantages of Spring's Static Context Cache (reuse the same ApplicationContext on every test)

When we run tests on our development VMs, using Oracle JDK 7u79/80 we see very fast tests. For example, a fairly big test class with 50+ test methods takes something around 1:30m (including context initialisation time) to run.

Our basic Dev VM is something like:

• Dual-Core i5 processor (with HyperThreading, so 4 virtual threads)
• 8GB RAM
• Mid Range, SATA HDD (5k or 7k2 RPM)
• Windows 7 x64 running Oracle JDK on Client mode (-client)

We use Jenkins as our CI server and Maven (3.2) takes care of the build process. We have a Master + 2 Slaves architecture. All the VMs are exactly the same:

• 8 Xeon E5 Cores (Real cores)
• 8 GB RAM
• SSD LUN serving the VMs (1.2GB/s mean throughput)
• Debian Linux 8 x64
• Oracle JDK on Server mode

Remember those tests that usually take 1:30 min to run on our Eclipse's? Here on the server they take more than 15 minutes to run! Here's what I already tried to do (without success so far.):

• Set MAVEN_OPTS with fixed heap size (2GB heap), large PermGen space, tuned GC settings, /dev/urandom as random seed
• Installed a 32 bits JDK to use the same Client mode we use on the Dev Machines
• Tuned on Surefire's configuration to increase memory, tune GC, etc. (As I'm not using a forked VM to run the tests anymore I took it out because it wouldn't change anything.)

Bottom line, is there any concrete reason on why Surefire's execution is s much slower than running JUnit on Eclipse? I'm scratching my head over this for a few days now and it's really starting to annoy me! I hate it when the solution seems to be so close but yet it is so far.

I couldn't test with Maven on my Dev machine because I couldn't allocate all the memory necessary to do it, but running chunks of classes (not the whole 1.8k tests suite) still reveals to be quite slower than running on Eclipse.

I understand Maven has all those phases and so on, but the difference should't be this big. Don't you agree?

Any input will be highly appreciated. I can provide any more information you find necessary!

PS: Surefire v2.17, Maven 3.2.2, JUnit 4.12, Spring Test 3.2.13

Thanks a lot!

UPDATE 1

I tried deactivating JaCoCo on the CI server to see if it affected the build times. It doesn't. The execution times remain the same.

Answer 1

I have made some progress on this. As I said above, this is a multi-modular (20+) Maven application. We keep all our tests separated on one module, but Surefire's configuration were being done on the parent POM, rather on the test module's POM.

After trying everything we decided to bring the test execution configuration to this module alone, skipping all the others. This brought the suite execution from ~45' to 22'. We're scratching our heads over this, and once we understand what is going on I'll post here again.

TL;DR: This image explains everything. (off-topic, safe for work)

Thanks for all your input!

Answer 2

You can try to reproduce the remote surefire runs by running mvn test locally. Are you sure that you are running with the same settings, regarding forking?

I know the fork settings of surefire are quite complicated and changed a bit over the time (see here for a detailed desciption: https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html).

If the tests get forked each time and don't reuse forks, the SpringTestRunner might spend a long time on initializing the application again and again.

Are you sure that it's the surefire tests that really slow things down?

Answer 3

I know this is an old thread, but maybe its still interesting: If your workload uses much stdout or stderr, surefire will create three times as big arrays as your output is, in order to save your unescaped output into them. This causes surefire to take much more time than executing the same test in eclipse or gradle.

I created a fork https://github.com/DaGeRe/maven-surefire and will see whether it is possible to merge this into surefire. My personal tests went from ~1100 ms to ~370ms with the patched plugin.