I have a CMake project with x86 native (win32) tests using CTest. I succesfully ran them with coverage results from Visual Studio Enterprise 2019 IDE using the Test Explorer.
Now I’m trying to run them from the command line. So far I have found that I should use vstest.console.exe, that I can provide settings through a *.runsettings file, but it is failing to discover my tests. I’m running it from my target/bin folder where the test exes are located with this command line:
vstest.console.exe char*test.exe /EnableCodeCoverage /Settings:..\..\CodeCoverage.runsettings
Output is:
Microsoft (R) Test Execution Command Line Tool Version 16.9.1
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in B:\MyApp\target_win10_msvc15_md\bin\charactersdefinitionelementtest.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Attachments:
B:\MyApp\TestResults\ae4d3abe-b935-47ef-b87a-5d43923fc4d9\[…]_2023-10-24.16_17_51.coverage
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:<pathToCustomAdapters>.
Output is the same for each of my test exe.
I suppose I have to register a test discoverer and/or executor but I can not found how and which one on the web and in Microsoft documentation:
- VSTest doc is silent on CTest and say nothing on registering a test discoverer or executor,
- .runsettings file doc is silent too on these,
- CTest in VS says nothing on using the command line.
My CodeCoverage.runsettings file:
<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<!-- Use 0 for maximum process-level parallelization. This does not force parallelization within the test DLL (on the thread-level). You can also change it from the Test menu; choose "Run tests in parallel". Unchecked = 1 (only 1), checked = 0 (max). -->
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to directory that contains .runsettings file-->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<!-- Omit the whole tag for auto-detection. -->
<!-- [x86] or x64, ARM, ARM64, s390x -->
<!-- You can also change it from the Test menu; choose "Processor Architecture for AnyCPU Projects" -->
<TargetPlatform>x86</TargetPlatform>
<!-- TestSessionTimeout was introduced in Visual Studio 2017 version 15.5 -->
<!-- Specify timeout in milliseconds. A valid value should be greater than 0 -->
<TestSessionTimeout>100000</TestSessionTimeout>
<!-- true or false -->
<!-- Value that specifies the exit code when no tests are discovered -->
<TreatNoTestsAsError>true</TreatNoTestsAsError>
</RunConfiguration>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<!-- BOOST -->
<ModulePath>boost_.*\.dll$</ModulePath>
<!-- XERCES -->
<ModulePath>xerces.*\.dll$</ModulePath>
</Exclude>
</ModulePaths>
<!-- Match fully qualified names of functions: -->
<!-- (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.) -->
<Functions>
<Exclude>
<Function>^boost::.*</Function>
<Function>^std::.*</Function>
<Function>^ATL::.*</Function>
<Function>.*::__GetTestMethodInfo.*</Function>
<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
</Exclude>
</Functions>
<!-- Match attributes on any code element: -->
<Attributes>
<Exclude>
<!-- Don't forget "Attribute" at the end of the name -->
<Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
<Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
<Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
<Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
</Exclude>
</Attributes>
<!-- Match the path of the source files in which each method is defined: -->
<Sources>
<Exclude>
<Source>.*\\atlmfc\\.*</Source>
<Source>.*\\vc\\tools\\.*</Source>
<Source>.*\\vctools\\.*</Source>
<Source>.*\\public\\sdk\\.*</Source>
<Source>.*\\microsoft sdks\\.*</Source>
<Source>.*\\vc\\include\\.*</Source>
</Exclude>
</Sources>
<!-- Match the company name property in the assembly: -->
<CompanyNames>
<Exclude>
<CompanyName>.*microsoft.*</CompanyName>
</Exclude>
</CompanyNames>
<!-- Match the public key token of a signed assembly: -->
<PublicKeyTokens>
<!-- Exclude Visual Studio extensions: -->
<Exclude>
<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
</Exclude>
</PublicKeyTokens>
<!-- We recommend you do not change the following values: -->
<!-- Set this to True to collect coverage information for functions marked with the "SecuritySafeCritical" attribute. Instead of writing directly into a memory location from such functions, code coverage inserts a probe that redirects to another function, which in turns writes into memory. -->
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<!-- When set to True, collects coverage information from child processes that are launched with low-level ACLs, for example, UWP apps. -->
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<!-- When set to True, collects coverage information from child processes that are launched by test or production code. -->
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<!-- When set to True, restarts the IIS process and collects coverage information from it. -->
<CollectAspDotNet>False</CollectAspDotNet>
<!-- When set to True, static native instrumentation will be enabled. -->
<EnableStaticNativeInstrumentation>True</EnableStaticNativeInstrumentation>
<!-- When set to True, dynamic native instrumentation will be enabled. -->
<EnableDynamicNativeInstrumentation>True</EnableDynamicNativeInstrumentation>
<!-- When set to True, instrumented binaries on disk are removed and original files are restored. -->
<EnableStaticNativeInstrumentationRestore>True</EnableStaticNativeInstrumentationRestore>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
How can I run my tests with code coverage from the command line?