This question is regarding the jumble mutation testing framework (http://jumble.sourceforge.net/)
i am trying to get jumble into the projects i work on my job. i've run jumble on the classes of one of this projects and it performs the mutations on some classes but for some others it tells
"test class is broken"
despite i run those test classes with an ant task and they run ok.
I've trying to know what is going wrong and started to run jumble in debug mode on eclipse for each one of the classes that jumble reported their tests to be broken. When i debug these classes, i get into junit class TestResult's instance fields so i can see the description of the errors and failures that junit reports when jumble run the test class, because jumble tells me "tests class is broken" but it doesn't tell why it says the tests class brokes so i need to debug to know what is happening. The issues and error details i've observed that are the result of the failed tests jumble does are some of the followings:
(I cannot post certain pieces of code due to legal restrictions of the company i work for, but i will try to be the more explanatory that i can)
For the class "Forwarder" i run jumble in debug mode and runs smoothly and makes the mutations perfectly. This is the case that surprises me the most, because this when run with the ant task jumble reports me "test class is broken", but in debug mode in eclipse runs ok.
For the class Action, and many others, in the debug mode an error occurs when the test class are run and when inspected the errors descriptions, the error is
java.lang.ClassNotFoundException: org.junit.rules.ExpectedException.
I have that class on the junit jar and it's path is available for the eclipse debug mode and for the ant task. Also, this class is in a jar of the selenium framework with its dependencies I pressume that it could be that the class loader can't decide what version of the class to load. But i'm not sure.
For the class Module, i do the same thing: inspect the error message that is available on the TestResult junit class when the tests is run by jumble and finishes executing. For this class i get:
java.lang.LinkageError: loading constraint violation: loader "com/reeltwo/jumble/mutation/MutatingClassLoader@5b8d5b8d" previously initiated loading for a different type with name "org/hamcrest/Description" defined by loader "sun/misc/Launcher$AppClassLoader@7fce7fce"
This error got me astonished. It makes me think that Jumble has its own class loader and maybe it somehow can be the source of some of my troubles.
I really will appreciate any kind of help, Marcos
Jumble uses a custom classloader to insert mutations, which may be the cause of some of your problems.
It also embeds its own version of junit which might be the cause of your classnotfound exceptions. Without a lot more information it's difficult to say for sure.
You could try asking the developers via the Jumble mailing list, or alternatively give PIT a try instead http://pitest.org.
PIT inserts mutations using the instrumentation API instead of using classloaders, does not embed JUnit and is generally a lot faster. I wrote it so am probably very biased however.