Failed Test does not fail task in YAML

Question

I have a build pipeline as shown below

    variables:
    - name: BuildParameters.solution
      value: '**/*.sln'
    - name: buildConfiguration
      value: Release
    name: $(date:yyyyMMdd)$(rev:.r)
    trigger: none
    resources:
      repositories:
      - repository: self
        type: git
    stages:
    - stage: __default
      jobs:
      - job: Job_1
        displayName: Agent job 1
        pool:
          vmImage: windows-2019
        steps:
        - task: UseDotNet@2
          displayName: '.NET Core 3.1.404'
          inputs:
            version: '3.1.404'
            packageType: sdk
        - task: DotNetCoreCLI@2
          enabled: true
          displayName: dotnet test
          continueOnError: true
          timeoutInMinutes: 0
          inputs:
            command: test
            projects: '**/*test/*.csproj'
            arguments: --configuration $(buildConfiguration) --verbosity n --collect "Code coverage"
            publishTestResults: true        
        - task: PublishTestResults@2
          inputs:
            testResultsFormat: XUnit
            testResultsFiles: '**/TEST.xml'
            mergeTestResults: true
            failTaskOnFailedTests: true

I have broken a test on purpose

This test fails, but the publish test results task doesnt fail outright

I get the warning

##[warning]No test result files matching **/TEST.xml were found. xunit

Has anyone ever had this?

Annoyingly if a test fails the Azure pipeline classes this as a partial success which is not ideal. I am trying to get into a situation where if a test fails, my pipeline fails

This pipeline is used to verify pull requests - i.e. do a build of the code in the PR and ensure that all tests pass

When I searched for this there were references to installing .NET 4.6.2 or later, but this is being used inside a build pipeline, so not sure how to do that

Have I missed something in the dot net test task?

Paul

Answer 1

When you set publishTestResults field to true for dotnet test task. See below extract from here.

This option appends --logger trx --results-directory $(Agent.TempDirectory) to the command line arguments.

Above commands will generate the test result in VSTest(trx) format and output the result to folder $(Agent.TempDirectory)

So you need to change the PublishTestResults@2 task like below:

 - task: PublishTestResults@2
         inputs:
           testResultsFormat: VSTest  #change the format to VSTest
           testResultsFiles: '**/*.trx'  #change to find .trx file
           mergeTestResults: true
           failTaskOnFailedTests: true
           searchFolder: '$(Agent.TempDirectory)' #search the .trx file in $(Agent.TempDirectory)

Usually the continueOnError field is set to false by default. And the dotnet test task will fail if there is any test fails, See below:

If the continueOnError field is set to true. The dotnet test task and the pipeline will show Warnig state.

So you can have a try setting the continueOnError field to false, which will fail the task and pipeline if there is any test fails. And there will not be no need to use PublishTestResults task. For publishTestResults is set to true for dotnet test task, the test result will be automatically published by the dotnet test task. See below:

 - task: DotNetCoreCLI@2
          enabled: true
          displayName: dotnet test
          continueOnError: false
          timeoutInMinutes: 0
          inputs:
            command: test
            projects: '**/*test/*.csproj'
            arguments: --configuration $(buildConfiguration) --verbosity n --collect "Code coverage"
            publishTestResults: true