Simple mutation unit testing erroneously report survival of the mutation of the exception message

Question

Suppose we have a Calculator with a divide method throwing error if the denominator is 0:

    public class Calculator
    {
        public double Divide(int x, int y)
        {
            if (y == 0)
            {
                throw new ArgumentOutOfRangeException(paramName: nameof(y), message: CalculatorErrorMessages.DenominatorShouldNotBeZero);
            }

            return (double)x / y;
        }
    }

    public static class CalculatorErrorMessages
    {
        public static string DenominatorShouldNotBeZero = "Denominator should not be zero.";
    }

And here's the unit test which tries to test the raise of the exception:

    public class CalculatorUnitTest
    {
        [Fact]
        public void Test1()
        {
            var calculator = new Calculator();

            var ex = Assert.Throws<ArgumentOutOfRangeException>(() => calculator.Divide(1, 0));
            Assert.Contains(CalculatorErrorMessages.DenominatorShouldNotBeZero, ex.Message);
        }
    }

Stryker.NET reports the exception message survives mutation:

But that mutation survival is not an actual problem, since both the unit test code and the app code generate the exception message in the same way. I don't want to duplicate and maintain error messages in the app and the unit test.

For this, I could make Stryker ignore mutations on CalculatorErrorMessages but it does not sound like a good idea. Stryker will not catch a situation where the unit test does not handle error messages like it should. For example, let's say I add a new method DivideV2 in the Calculator which also throws an exception, and in the unit-test, instead of using the actual error message constant, I hard-code the expected error message:

        [Fact]
        public void Test2()
        {
            var calculator = new Calculator();

            var ex = Assert.Throws<ArgumentOutOfRangeException>(() => calculator.DivideV2(1, 0));
            Assert.Contains("Denominator should not be zero.", ex.Message);
        }

In this situation, the mutation survival should really not be ignored.

How can I make in such a way Stryker:

• Does not show me 'false' mutations survivals
• Still make my code unit tested by mutations
• Not duplicate the error message strings across the app and unit-test

Answer 1

So, long story short, one way to achieve your goals is to make the string const. Keep in mind that this way you are making use of the Stryker limitation described here. So different mutation testing framework or maybe some future Stryker version can still mark this as a surviving mutant.

Also, keep in mind that const is not "better" than static and replacing one with another can have unpleasant consequences. You shouldn't make it const just to calm Stryker down. In your case, I'd probably have it static readonly.

Ultimately, you need to define the level of testing for yourself, how black/white box you want it to be, how tautological you want it to be. And remember, you can ignore specific mutations.

Answer 2

Mutation vs Unit tests

To reiterate their purpose:

• Unit tests are used to check your code for errors when changes to code are applied
• Mutation tests are used to check if changes to your code, actually trip up the unit test. It does so by modifying the actual code and re-running tests, expecting them to fail.

The Stryker docs explain what mutations will be applied to the code.
In this case, a public static string YourVariable = "your-value"
would be change to public static string YourVariable = "", expecting your unit test to fail.

Calculator example:

You've expressed and identified couple of concerns:

I don't want to duplicate and maintain error messages in the app and the unit test.

This makes total sense to me. In my opinion moving in this direction would leave you with very brittle tests.

Ignore mutations on CalculatorErrorMessages but it does not sound like a good idea.

That's also correct, for the same reason you gave as an example. you want the mutation test to flag your code as properly tested when, in the unit test, you provide a hardcoded expectation on the error message.

your goals:

• Stryker does not show me 'false' mutations survivals. Is caused by the test not failing when a mutation is done.
• Still make my code unit tested by mutations. Is done by not ignoring stryker mutation on DenominatorShouldNotBeZero
• Not duplicate the error message strings across the app and unit-test. Is done by writing the test like you did in the first example AND making sure mutations actually fail your unit tests.

That brings us to the crux, we have to make sure the unit-test fails after mutation (aka, killing the mutation). You could add additional tests & conditions that will trip after mutation.

What are we testing

we want the actual code:

• To not throw exception and correctly calculate the division
• To throw an exception when it's 0
• The exception type to be of type ArgumentOutOfRangeException
• The exception to use some constant as message
• The message not to be empty
• The exception to have paramName, (equal to the 2nd param name, or just not empty)

we want the unit test to test all of the above.

The mutation test is has correctly identified that a change on DenominatorShouldNotBeZero is currently not tripping the test. Changing the const to '', will still pass the unit-test.
It expects the unit test to fail when someone would change your const to ''.

I could assert that ex.Message is not an empty string.