I'm currently running into the following problem:
We introduced Unit-Tests into our native c++ application by using the native microsoft unit test framework included in Visual Studio 2015 Microsoft::VisualStudio::CppUnitTestFramework.
To start the unit tests inside visual studio we use the Test-Explorer as can be seen in image 1. By adding a method attribute to a Test-Method (as shown in image 2) the Test-Explorer allows to sort and filter by attribute (as can be seen in image 3).
All this works without a problem. We started having a problem when we tried to implement these unit tests into our continous integration system (CI).
To start Unit-Tests from a console we use vstest.console.exe, which is located in: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe. The official documentation can be found Here.
The following screenshot shows how the tests succeed when using the console:
Adding a logger and outputing files to .trx works well too. What does NOT work however is applying a filter to determine tests with specific attributes. According to the documentation something like that should be possible by using the /TestCaseFilter:-switch. The following screenshot however shows the resulting error:
It translates to: "No tests matched the filter because it contains one or more properties that are not valid (TakesLong). Specify filter expression containing valid properties (TestCategory, Priority, FullyQualifiedName, Name) and try again."
Does anybody know this problem or has an idea on how to fix it? Ultimate goal is to be able to exclude certain tests (Take very long / need a hardware dongle etc.).
Best regards, LorToso
I ran into this problem too, and sadly it looks like we're just out of luck. vstest.console.exe only supports a limited number of filters for both C++ and C#.
The approach I think I'm going to move towards is to segregate tests into different modules so I can run unit tests for nightly and CI builds and integration tests (like needing a database) only for releases.
Alternatively, you could split them up by namespace in the same module and use the FullyQualifiedName filter. Unfortunately, there is no '!~' condition either so you might have to work around that somehow.