Any way to link published test results with the test plan test cases in Azure DevOps?

Question

I am trying to run automated tests using Cypress in Azure DevOps and publishing the tests results(Using PublishTestResult@2 Task) as an XML file using JUnit XML Reporter and merging them.

I am having trouble linking these published test results with the test cases in my ADO Test Plans, to my understanding JavaScript frameworks are currently not supported in Visual Studio to associate the tests with the test case ID, is there a workaround with this? Any help appreciated!

Tried using Visual Studio to associate the tests, publishing the results as an artifact to see if there was a way.

Answer 1

According to the document to Associate automated tests with test cases - Azure Test Plans | Microsoft Learn, please check if the types of tests are supported.

Coded UI test, Selenium tests, and unit tests written using Version 1 of the MSTest framework can be associated with a test case.

Tests that use MSTest v2, NUnit, and xUnit frameworks can be associated with a test case workitem when using Visual Studio 15.9 Preview 2 or later. However, these tests cannot be run using Microsoft Test Manager and XAML builds.

Tests that use the .NET core framework can be associated with a test case workitem when using Visual Studio 15.9 Preview 2 or later. To run the .NET core tests the appropriate target framework must be specified in a runsettings file. However, these tests cannot be run using Microsoft Test Manager and XAML builds.

Tests that use other test frameworks such as Chutzpah (for JavaScript tests such as Mocha or QUnit), or Jest cannot be associated with a test case.

Associating generic tests may work, but running these tests is not supported.

If our project is not using any of the supported framework and we cannot associate automated tests with test cases in a test plan, we are unable to Run automated tests from test plans because the release pipeline will need to know which accembly to test based on the link associated to the test case.

For this, you may raise a feature request, that will allow you to directly interact with the appropriate product group, and make it more convenient for the product group to collect and categorize your suggestions.

Hope the information would help.

Answer 2

This question has come up before.

The short answer is it should be possible through automation.

The long answer is the test results that are produced from your build pipeline and the test results that are associated with a Test Plan are slightly different and you need a mechanism to map between them.

When looking at Test Plans in Azure DevOps, it's important to recognize:

• A Test Plan has a set of Test Suites
• A Test Suite has a set of Test Cases
• Test Cases are reusable, but when you carry a Suite to another Plan it creates a copy of the original Test Suite.
• A Test Plan has a set of Test Configurations. These can represent the various targets that your test cases will be executed against, typically Operating Systems, Web Browsers or combinations of the two. Even if you're not explicitly using them, they are required, so you need at least one.
• While it might seem like you're simply executing a Test Case, you are actually executing a combination of the Test Plan + Test Case + Test Configuration. These three things taken together are referred to as a "Test Point".
• Reporting for Test Plans is done by outcomes for Test Points, not Test Cases.

You need a mapping between the Test Points and your automated tests.

How Automated Test Cases Work

• There is meta-data in each Test Case that represents a pointer to the test-automation. These fields are read-only in the user-interface and can only be set through Visual Studio (or through the REST API).
• The feature in Visual Studio that performs the "linking" is a mechanism that puts the predictable, fully qualified test name (Namespace + Class Name + Method + dll Assembly) into the meta-data of the Test Case. It's because these values are predictable is why this feature is only supported for .NET.
• The feature in Azure DevOps (VSTest@2) that runs the automated tests and publishes them to the Test Plan requires the Assembly, the Test Plan identifier, and Test Configuration identifier as inputs. It takes the test results (Namespace+Class+Method,Assembly) and maps them to the Test Cases to obtain the Test Points.

How you can map your Test Results

The challenge with creating a universally supported approach outside of the .NET ecosystem is the reliable mapping of the test name in the JUnit/NUnit/xUnit test result to the Test Case. JavaScript, Java and Cucumber frameworks are horribly inconsistent and can vary per implementation, so it ultimately requires that you, the developer, are responsible for establishing that mapping. If you can produce that mapping, the rest is automation against the REST APIs:

1. Locate the Test Plan id
2. Locate the Configuration Id
3. Locate the Test Suites in the Test Plan
4. Locate the Test Points in the Suite + Configuration
   • Correlate the Test Point to the Test Case ID
5. Create a Test Run for these Test Points
6. Obtain the Test Result for the Test Run
7. Patch the Test Result with the outcomes from your automated test result

There is an article that describes the above here.

One approach to provide the mapping of Test Case to test result, is to put the ID of the Test Case in the title of the test, eg <TestCase-ID>_<TestName>.

Update 2024/01/30:

I have an Azure DevOps Extension that can perform the above. If you'd like to be added to the private-preview, please submit a request here