Nunit order of execution using the TestCase attribute

Question

I'm trying to run some performance tests using the NUnit framework, e.g.:

[TestCase(1)]
[TestCase(10)]
[TestCase(100)]
[TestCase(1000)]
public void SaveBookingTest(int numberOfItems)
{
    var stopwatch = new Stopwatch();

    var items = DataFactory.CreateItems(numberOfItems);

    stopwatch.Start();
    service.SaveItems(items);
    stopwatch.Stop();

    SaveResults(stopwatch.ElapsedMilliseconds);
}

... and the result is:

1. 22:56:01: Saving 1000 items in database: 48353ms - Avg.: 20,6812400321 items/sec.
2. 22:56:02: Saving 1 items in database: 40ms - Avg.: 25 items/sec.
3. 22:56:06: Saving 100 items in database: 3813ms - Avg.: 26,2260687323 items/sec.
4. 22:56:07: Saving 10 items in database: 413ms - Avg: 24,2130750605 items/sec.

So what I would expect is that [TestCase(1)] would be run first, then [TestCase(10)] and so on. But as you can see from the results, this does not happen. Is there anyway to control this? It seems to be sort of random which testcases finishes first.

According to NUnit documentation, I guess the answer is that you can't.

As a result, when TestCaseAttribute appears multiple times on a method or when other data-providing attributes are used in combination with TestCaseAttribute, the order of the test cases is undefined.

Have you guys any good ideas how to control the execution? Small hacks? Another way of doing it? Anything?

Answer 1

Did you consider using TestCaseSourceAttribute instead? Or is it not applicable to you particular situation? Interesting part from documentation is:

However, when a single TestCaseSourceAttribute is used by itself, the order of the tests follows exactly the order in which the test cases are returned from the source.

The modified code would look like:

[Test, TestCaseSource("GetTestCaseSource")]
public void SaveBookingTest(int numberOfItems)
{
    var stopwatch = new Stopwatch();

    var items = DataFactory.CreateItems(numberOfItems);

    stopwatch.Start();
    service.SaveItems(items);
    stopwatch.Stop();

    SaveResults(stopwatch.ElapsedMilliseconds);
}

public IEnumerable<int> GetTestCaseSource()
{
    return new[] { 1, 10, 100, 1000 };
}