How to run CTest with coverage by Visual Studio 2019 from the command line?

79 views Asked by At

I have a CMake project with x86 native (win32) tests using CTest. I succesfully ran them with coverage results from Visual Studio Enterprise 2019 IDE using the Test Explorer.

Now I’m trying to run them from the command line. So far I have found that I should use vstest.console.exe, that I can provide settings through a *.runsettings file, but it is failing to discover my tests. I’m running it from my target/bin folder where the test exes are located with this command line:

vstest.console.exe char*test.exe /EnableCodeCoverage /Settings:..\..\CodeCoverage.runsettings

Output is:

Microsoft (R) Test Execution Command Line Tool Version 16.9.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in B:\MyApp\target_win10_msvc15_md\bin\charactersdefinitionelementtest.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Attachments:
  B:\MyApp\TestResults\ae4d3abe-b935-47ef-b87a-5d43923fc4d9\[…]_2023-10-24.16_17_51.coverage
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Output is the same for each of my test exe.

I suppose I have to register a test discoverer and/or executor but I can not found how and which one on the web and in Microsoft documentation:

My CodeCoverage.runsettings file:

<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
    <!-- Configurations that affect the Test Framework -->
    <RunConfiguration>
        <!-- Use 0 for maximum process-level parallelization. This does not force parallelization within the test DLL (on the thread-level). You can also change it from the Test menu; choose "Run tests in parallel". Unchecked = 1 (only 1), checked = 0 (max). -->
        <MaxCpuCount>1</MaxCpuCount>
        <!-- Path relative to directory that contains .runsettings file-->
        <ResultsDirectory>.\TestResults</ResultsDirectory>

        <!-- Omit the whole tag for auto-detection. -->
        <!-- [x86] or x64, ARM, ARM64, s390x  -->
        <!-- You can also change it from the Test menu; choose "Processor Architecture for AnyCPU Projects" -->
        <TargetPlatform>x86</TargetPlatform>

        <!-- TestSessionTimeout was introduced in Visual Studio 2017 version 15.5 -->
        <!-- Specify timeout in milliseconds. A valid value should be greater than 0 -->
        <TestSessionTimeout>100000</TestSessionTimeout>

        <!-- true or false -->
        <!-- Value that specifies the exit code when no tests are discovered -->
        <TreatNoTestsAsError>true</TreatNoTestsAsError>
    </RunConfiguration>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <CodeCoverage>
                        <!-- Match assembly file paths: -->
                        <ModulePaths>
                            <Include>
                                <ModulePath>.*\.dll$</ModulePath>
                                <ModulePath>.*\.exe$</ModulePath>
                            </Include>
                            <Exclude>
                                <!-- BOOST -->
                                <ModulePath>boost_.*\.dll$</ModulePath>
                                <!-- XERCES -->
                                <ModulePath>xerces.*\.dll$</ModulePath>
                            </Exclude>
                        </ModulePaths>

                        <!-- Match fully qualified names of functions: -->
                        <!-- (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.)  -->
                        <Functions>
                            <Exclude>
                                <Function>^boost::.*</Function>
                                <Function>^std::.*</Function>
                                <Function>^ATL::.*</Function>
                                <Function>.*::__GetTestMethodInfo.*</Function>
                                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
                            </Exclude>
                        </Functions>

                        <!-- Match attributes on any code element: -->
                        <Attributes>
                            <Exclude>
                                <!-- Don't forget "Attribute" at the end of the name -->
                                <Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
                                <Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
                                <Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
                                <Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
                            </Exclude>
                        </Attributes>

                        <!-- Match the path of the source files in which each method is defined: -->
                        <Sources>
                            <Exclude>
                                <Source>.*\\atlmfc\\.*</Source>
                                <Source>.*\\vc\\tools\\.*</Source>
                                <Source>.*\\vctools\\.*</Source>
                                <Source>.*\\public\\sdk\\.*</Source>
                                <Source>.*\\microsoft sdks\\.*</Source>
                                <Source>.*\\vc\\include\\.*</Source>
                            </Exclude>
                        </Sources>

                        <!-- Match the company name property in the assembly: -->
                        <CompanyNames>
                            <Exclude>
                                <CompanyName>.*microsoft.*</CompanyName>
                            </Exclude>
                        </CompanyNames>

                        <!-- Match the public key token of a signed assembly: -->
                        <PublicKeyTokens>
                            <!-- Exclude Visual Studio extensions: -->
                            <Exclude>
                                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
                            </Exclude>
                        </PublicKeyTokens>

                        <!-- We recommend you do not change the following values: -->

                        <!-- Set this to True to collect coverage information for functions marked with the "SecuritySafeCritical" attribute. Instead of writing directly into a memory location from such functions, code coverage inserts a probe that redirects to another function, which in turns writes into memory. -->
                        <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
                        <!-- When set to True, collects coverage information from child processes that are launched with low-level ACLs, for example, UWP apps. -->
                        <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
                        <!-- When set to True, collects coverage information from child processes that are launched by test or production code. -->
                        <CollectFromChildProcesses>True</CollectFromChildProcesses>
                        <!-- When set to True, restarts the IIS process and collects coverage information from it. -->
                        <CollectAspDotNet>False</CollectAspDotNet>
                        <!-- When set to True, static native instrumentation will be enabled. -->
                        <EnableStaticNativeInstrumentation>True</EnableStaticNativeInstrumentation>
                        <!-- When set to True, dynamic native instrumentation will be enabled. -->
                        <EnableDynamicNativeInstrumentation>True</EnableDynamicNativeInstrumentation>
                        <!-- When set to True, instrumented binaries on disk are removed and original files are restored. -->
                        <EnableStaticNativeInstrumentationRestore>True</EnableStaticNativeInstrumentationRestore>

                    </CodeCoverage>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

How can I run my tests with code coverage from the command line?

0

There are 0 answers