General goal
I am trying to create a PowerShell script that will run all the tests in an entire solution headless (via commandline) for CI purposes. We have UnitTests and IntegrationTests which should be able to run separately.
Solution Structure
The solution is a Visual Studio 2013 solution (using the Professional version). Every project has optionally one or two separate projects for testing. One project is for unit tests and the other project is for integration tests. We follow a simple naming convention:
- The component : Namespace.ComponentName
- The Unit test : Namespace.ComponentName.Test
- The Int. test : Namespace.ComponentName.IntegrationTest
Test Tools
I am using VSTest.Console.exe to run the tests.
Current failing solution
I am trying to run OR only unit tests OR only the integration tests. So the idea was to scan for all the test dll files and filter them on the suffix.
An example for the Unit Tests :
# Declare my parameter that will contain the targeted TestFileNames
$Parameters = '';
# Get all the test project names, space separated that match *.Test.dll in my solution
Get-ChildItem $RepositoryRootLocation -Filter *.Test.dll -Recurse | ? {$_.fullname -notmatch 'obj'} | % { $Parameters = $Parameters + $_.FullName+ " "};
#Run the VSTest.Console.Exe with the specified test project names
& $MsTestConsolePath $Parameters ;
This should work fine, the only problem is that I have 40+ test projects in nested subfolders. This causes an exception when running this script :
Get-ChildItem : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
The message is self explanatory, so the issue is, this issue is not scaleable since I am hitting some annoying simple limitations.
The main question
Is there another way to use VSTest.Console.exe that I can run only a certain subset of test projects based on a filter so I can run only OR the unit tests OR the integration tests.
Should I take another angle to tackle the problem ? Or can I just live with only the test project names instead of the full file paths when all test projects are in subfolders of my solution repo ?
Important : It should scale easily, so I don't need for every newly created test project to add this to a testsettings file or something a like. I want to be able to just say, run all tests projects in the solution, filtering if they are of type Unit or Integration.
Having my repo in the root of my disk to shorten the full path names of the test project names is not flexible since the script might run on any machine.
I would try several things.
First, change the path of your build server to a folder at the root of a drive. Less folders, less chance to run into the "path too long" problem.
You say that shortening the path might not be the right solution, but it is. You can have requirements on your build machines ("have .net installed" is one, for instance), and "not put the sources in a crazy sub folder" is a sane requirement.
Second, you can change the output path of your projects to another folder. For instance, create a "bin" folder at the root of your solution, and "debug", "release" and "test" sub-folders.
Then, configure your projects to output their binaries at the root, depending on their type and configuration (the regular projects to bin\debug|release ; and the test projects to bin\test regardless of debug or release).
You may not want to configure "each newly created test project" like this, but realistically, how many new projects will you create once your solution becomes a bit more stable?
Configuring the paths for your projects is a common devops/continuous integration task. Don't fool yourself, don't try to be overly clever, and don't assume your build script should run anywhere in any case. When your project has any kind of complexity (relying on external dependencies), you will have to define build requirements like "have this and that installed", so why not "have the sources maximum 2 or 3 folders deep from the disk root"?.