Can nunit-console list all test names in a test fixture?

Question

I'd like to report them before they're run, and have the option to run individual tests through shell scripts without managing categories. We have some unmanaged code which can leave the process in a bad state, and sometimes are happy to run each test individually per nunit-console run.

Answer 1

There is now the --explore command-line option that can be used to list all test cases without running tests. More specifically

nunit3-console.exe MyProject.dll --explore

For more info: https://github.com/nunit/docs/wiki/Console-Command-Line#description

Answer 2

nunit-console still doesn't seem to support this option. It is however fairly straightforward to get a list of the test cases using reflection. At a very basic level, you want a list of all public methods of any public class with the appropriate [Test]/[TestFixture] attributes. Depending upon how your tests are structured, you might need to do additional filtering, such as to remove any tests marked up with [Ignore] attributes or to consider test methods in base classes.

At a basic level, the code would look something like this:

// Load the assembly containing your fixtures
Assembly a = Assembly.LoadFrom(assemblyName);

// Foreach public class that is a TestFixture and not Ignored
foreach (var c in a.GetTypes()
                   .Where(x=>x.IsPublic 
                   && (x.GetCustomAttributes(typeof(NUnit.Framework.TestFixtureAttribute)).Count() > 0)
                   && (x.GetCustomAttributes(typeof(NUnit.Framework.IgnoreAttribute)).Count() ==0))) 
{
    // For each public method that is a Test and not Ignored
    foreach (var m in c.GetMethods()
                       .Where(x=>x.IsPublic
                       && (x.GetCustomAttributes(typeof(NUnit.Framework.TestAttribute)).Count() > 0)
                       && (x.GetCustomAttributes(typeof(NUnit.Framework.IgnoreAttribute)).Count() ==0))) 
    {
        // Print out the test name
        Console.WriteLine("{0}.{1}", c.ToString(), m.Name);
        // Alternately, print out the command line to run test case using nunit-console
        //Console.WriteLine("nunit-console /run:{0}.{1} {2}", c.ToString(), m.Name, assemblyName);
    }
}

Obviously you'd be able to streamline this a bit if you only wanted the test methods from a particular TestFixture.

As has been said in the comments, this gets a bit more complicated if you need to pay attention to other NUnit attributes, such as TestCase and TestCaseSource. I've modified the code below to support some of the functionality of these attributes.

static void PrintTestNames(string assemblyName) {
    Assembly assembly = Assembly.LoadFrom(assemblyName);

    foreach (var fixture in assembly.GetTypes().Where(x => x.IsPublic
                                      && (x.GetCustomAttributes(typeof(TestFixtureAttribute)).Count() > 0)
                                      && (x.GetCustomAttributes(typeof(IgnoreAttribute)).Count() == 0))) {
        foreach(var method in fixture.GetMethods().Where(x=>x.IsPublic
            && (x.GetCustomAttributes(typeof(IgnoreAttribute)).Count() == 0)
            && ((x.GetCustomAttributes(typeof(TestAttribute)).Count() > 0)
                || (x.GetCustomAttributes(typeof(TestCaseAttribute)).Count() > 0)
                || (x.GetCustomAttributes(typeof(TestCaseSourceAttribute)).Count() > 0))
            )) {
            var testAttributes = method.GetCustomAttributes(typeof(TestAttribute)) as IEnumerable<TestAttribute>;
            var caseAttributes = method.GetCustomAttributes(typeof(TestCaseAttribute)) as IEnumerable<TestCaseAttribute>;
            var caseSourceAttributes = method.GetCustomAttributes(typeof(TestCaseSourceAttribute)) as IEnumerable<TestCaseSourceAttribute>;

            if (caseAttributes.Count() > 0) {
                foreach(var testCase in caseAttributes) {
                    if (!string.IsNullOrEmpty(testCase.TestName)) {
                        PrintTestName(fixture.ToString(), testCase.TestName);
                    }
                    else {
                        string arguments = ExtractArguments(testCase.Arguments);
                        PrintTestName(fixture.ToString(), method.Name + arguments);
                    }
                }
            }
            else if (caseSourceAttributes.Count() > 0) {
                foreach (var testCase in caseSourceAttributes) {
                    var sourceName = testCase.SourceName;
                    if (!string.IsNullOrEmpty(sourceName)) {
                        var staticMember = fixture.GetField(sourceName, BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public);
                        var instanceMember = fixture.GetField(sourceName, BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
                        var staticMethodMember = fixture.GetMethod(sourceName, BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public);
                        var instanceMethodMember = fixture.GetMethod(sourceName, BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
                        var staticPropMember = fixture.GetProperty(sourceName, BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public);
                        var instancePropMember = fixture.GetProperty(sourceName, BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);


                        IEnumerable memberValues;

                        if (null != staticMember) {
                            memberValues = staticMember.GetValue(null) as IEnumerable;
                        }
                        else if (null != instanceMember) {
                            var instance = Activator.CreateInstance(fixture);
                            memberValues = instanceMember.GetValue(instance) as IEnumerable;
                        } else if(null != staticMethodMember) {
                            memberValues = staticMethodMember.Invoke(null,new object [0]) as IEnumerable;
                        }
                        else if (null != instanceMethodMember) {
                            var instance = Activator.CreateInstance(fixture);
                            memberValues = instanceMethodMember.Invoke(instance, new object[0]) as IEnumerable;
                        }
                        else if (null != staticPropMember) {
                            memberValues = staticPropMember.GetValue(null) as IEnumerable;
                        }
                        else if (null != instancePropMember) {
                            var instance = Activator.CreateInstance(fixture);
                            memberValues = instancePropMember.GetValue(instance) as IEnumerable;
                        }
                        else {
                            Console.WriteLine("*** Ooops...Looks like I don't know how to get {0} for fixture {1}", sourceName, fixture.ToString());
                            continue;
                        }

                        foreach (var memberValue in memberValues) {
                            if (null != memberValue as IEnumerable) {
                                PrintTestName(fixture.ToString(), method.Name + ExtractArguments(memberValue as IEnumerable));
                            }
                            else {
                                PrintTestName(fixture.ToString(), method.Name + "(" + memberValue.ToString() + ")");
                            }
                        }
                    } else {
                        Console.WriteLine("*** Ooops...Looks like I don't know how to handle test {0} for fixture {1}", method.Name, fixture.ToString());
                    }
                }
            }
            else {
                PrintTestName(fixture.ToString(), method.Name);
            }
        }
    }
}

static string ExtractArguments(IEnumerable arguments) {
    string caseArgs = "(";
    bool first = true;
    foreach (var arg in arguments) {
        if (first) first = false;
        else caseArgs += ",";
        caseArgs += Convert.ToString(arg);
    }
    return caseArgs + ")";
}

static void PrintTestName(string fixture, string testName) {
    Console.WriteLine("{0}.{1}", fixture, testName);
    //Console.WriteLine("nunit-console /run:{0}.{1} {2}", fixture, testName, assemblyName);
}

If you look through the code above, you might notice that I've handled functionality where the TestCaseSource for tests is a string naming the property/method/field. You'll also notice that whilst there is more code, it's still pretty straightforward code, so it could be easily extended if you were using an alternative version of the TestCaseSource, or if there are other NUnit attributes you're using that I haven't catered for.

It would also be easy enough to add a counter to the above so that you had a level of comfort that the same number of tests were printed as the number of tests that nunit-console would be running.