Azure DevOps DotNetCoreCLI@2 test fails with coverlet total coverage

Question

My Azure DevOps pipeline is failing on the DotNetCoreCLI@2 step with error code 1, even if the total coverage is above the threshold. When executed locally the command runs without errors and in the logs of the pipeline there are no error marked lines.

My Pipeline Step:

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: 'Solution.sln'
    arguments: '--configuration Release /p:"CollectCoverage=true" /p:"CoverletOutput=$(Common.TestResultsDirectory)/coverage.json" /p:"MergeWith=$(Common.TestResultsDirectory)/coverage.json" /p:"Threshold=70" /p:"ThresholdStat=\"Total\"" /p:"Exclude=\"SomeExclusions\""'
    testRunTitle: '$(Build.DefinitionName).$(Build.BuildId)'

There are multiple test project in the solution which have the following coverages:

+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| Module1               | 69.34% | 58.12% | 67.25% |
+-----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 69.34% | 58.12% | 67.25% |
+---------+--------+--------+--------+
| Average | 69.34% | 58.12% | 67.25% |
+---------+--------+--------+--------+

This has these errors below which are not marked red as errors in DevOps log:

coverlet.msbuild.targets(72,5): error : The total line coverage is below the specified 70 [Test1.csproj]
coverlet.msbuild.targets(72,5): error : The total branch coverage is below the specified 70 [Test1.csproj]
coverlet.msbuild.targets(72,5): error : The total method coverage is below the specified 70 [Test1.csproj]

Second has following output:

+------------------------+--------+--------+--------+
| Module                 | Line   | Branch | Method |
+------------------------+--------+--------+--------+
| Module2                | 78%    | 70.31% | 70.31% |
+------------------------+--------+--------+--------+
| Module1                | 91.24% | 90%    | 95.9%  |
+------------------------+--------+--------+--------+
| Module3                | 84.37% | 100%   | 90%    |
+------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 79.94% | 75.98% | 79.44% |
+---------+--------+--------+--------+
| Average | 84.53% | 86.77% | 85.4%  |
+---------+--------+--------+--------+

No errors. After the last test project comes the following output:

+---------------------------+--------+--------+--------+
| Module                    | Line   | Branch | Method |
+---------------------------+--------+--------+--------+
| Module4                   | 100%   | 100%   | 100%   |
+---------------------------+--------+--------+--------+
| Module2                   | 78%    | 70.31% | 70.31% |
+---------------------------+--------+--------+--------+
| Module1                   | 91.24% | 90%    | 95.9%  |
+---------------------------+--------+--------+--------+
| Module3                   | 84.37% | 100%   | 90%    |
+---------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 79.94% | 75.98% | 79.44% |
+---------+--------+--------+--------+
| Average | 88.4%  | 90.07% | 89.05% |
+---------+--------+--------+--------+

For me resulting in coverage higher than 70% as configured, but the DevOps says:

##[error]Error: The process 'dotnet.exe' failed with exit code 1

Am I understanding the concept of /p:"ThresholdStat=\"Total\"" wrong? I mean it works like a charm locally. And if I lower the threshold below 58% it works on the DevOps. DotNET Version 7 and Agent Version 2.218.1.

EDIT:

It only looks like everything is working locally. If I echo the %errorlevel% it gives back 1, so the DevOps is working correctly, but coverlet seems to be wrong here.

Answer 1

The pipeline failed because the project [Test1.csproj] has the test result: Total | 69.34% | 58.12% | 67.25% and the total coverage is below the specified 70. This should be expected behavior.

When a project test is completed, the coverlet will evaluate whether the total coverage meets the Threshold you set, instead of not evaluating until all projects in *.sln have been tested.

I tested both on local and in the pipeline, and got the same error when the first project didn't meet the Threshold. If it works in your local, you can double check if the local CoverageResults folder is clean and try again.

My test screenshot in local:

My test screenshot in pipeline:

By the way, according to this example, you can run dotnet test and MergeWith single command from a solution file, but you need to ensure that tests will run sequentially by adding -m:1 to the command.