Incremental run of testsuite

Question

We have a large project that has several thousands of tests in the testsuite, and the full testsuite run takes very long time.

I am looking for a tool that I can integrate in the Maven build that will run only those tests that might be affected (knowing code coverage for each), because some covered code has changes.

I was googling that and found a few similar things but not a perfect fit:

• Ekstazi http://www.ekstazi.org/ looks like exactly that, but it does not work out-of-the box with TestNG (used in the testsuite), and it is not open source
• Infinitest https://infinitest.github.io/ seems to focus mainly on IDE integration - is it possible to run the tests only on demand (just like mvn infinitest)?
• PIT http://pitest.org/ is not exactly what I am looking for but it also needs to analyze per-test coverage

It would be also very useful to remember test coverage with (last) git commit and run the tests against the last code changes.

Further suggestions and comments on those above are welcome.

Answer 1

As far as I can see, Infinitest doesn't provide corresponding Maven plugin, so it's impossible to do using it. You may consider creating it though, making an invaluable contribution to the world.

As far as I can see, it provides pretty solid API so writing a plugin shouldn't be a big problem. You may want to take a look at InfinitestCore interface first. If you're using a CI environment you may want to provide file list for the Infinitest directly from git diff --name-only HEAD~1 which will produce the list of files changed in latest commit (as an example, if you run your builds against each commit).

UPD. It seems like there's a workaround involving maven-exec-plugin to explicitly run Infinitest in the Maven build: you can run 'mvn exec:exec' from the command line or from m2eclipse's Maven Build launcher to run Infinitest against your project. I'd advice specifying the explicit build phase on which it should be run using the executions element in POM:

executions: It is important to keep in mind that a plugin may have multiple goals. Each goal may have a separate configuration, possibly even binding a plugin's goal to a different phase altogether. executions configure the execution of a plugin's goals.