Why does Pester in Classic Azure Pipeline always fail?

Question

I have a classic build pipeline (not yaml, yet!) with three tasks

• Install Pester -> works as expected
• Run Pester tests -> always fails!
• Publish test results -> works as expected

My script for "Run Pester tests"

Import-Module Pester -Force

$config = [PesterConfiguration]::Default
$config.TestResult.Enabled = $true
$config.TestResult.OutputFormat = "NUnitXml"
$config.TestResult.OutputPath = "$env:SYSTEM_DEFAULTWORKINGDIRECTORY\TEST-results.xml"
$config.Run.Path = ".\my\path*"
$config.Run.PassThru = $true

$results = Invoke-Pester -Configuration $config
if($results.Result -eq "Passed")
{
    Write-host "All tests passed"
    Write-Host "##vso[task.complete result=Succeeded;]DONE"
    exit 0
}

Sample output in Azure DevOps is

Tests completed in 11.43s
Tests Passed: 67, Failed: 0, Skipped: 0 NotRun: 0
All tests passed
Finishing: Run Pester Unit Tests

However, the task always fails

I know from robocopy that Powershell Tasks in Azure will check $LASTEXITCODE and that you are able to overwrite that via Write-Host "##vso[task.complete result=Succeeded;]DONE" and exit 0. Why is this not working here?

What else have I tried?

• Activate SilentlyContinue -> results in Warning
• Activate Continue on Error -> results in Warning
• Activate Ignore $LASTEXITCODE -> results in Failure
• Activate Fail on Standard Error -> results in Failure

Answer 1

Oh boy this was a tough one: The comment by Shayki Abramczyk enabled me to track it down. Is it possible to grant him the bounty somehow?

The cause was a unit test, that was changing the Azure DevOps task output via Write-Host "##vso[task.complete result=Failed;]DONE". Since my Powershell scripts are supposed to work in Azure DevOps, they should do things like this. The test was named nicely for Pester it "Should print out error if build requirement not found"

My learning here

1. Make your issues smaller. Reducing this to a single test, showed me directly that one of my tests was the cause even though they all passed.
2. You cannot test the following line in Azure DevOps with Pester, because if will always make your tests fail Write-Host "##vso[task.complete result=Failed;]DONE"
3. A test executing that line, will show "Passed" which can be correct, but is annoying and hard to find.
4. You cannot overwrite that by the following line afterwards Write-Host "##vso[task.complete result=Succeeded;]DONE"

Answer 2

The relevant documentation is "Azure DevOps / Azure Pipelines / Logging commands / Complete: Finish timeline"

##vso[task.complete]current operation

result =

• Succeeded: The task succeeded.
• SucceededWithIssues: The task ran into problems. The build will be completed as partially succeeded at best.
• Failed: The build will be completed as failed. (If the Control Options: Continue on error option is selected, the build will be completed as partially succeeded at best.)

That means when you use the ##vso[task.complete] logging command in Azure DevOps, it directly controls the outcome of the task. If your unit test executes a line that sets the task result to Failed, then the task will fail, even if all your tests pass.

To prevent this, you might want to consider a different approach to testing that line of code. Instead of actually running the Write-Host "##vso[task.complete result=Failed;]DONE" line in your test, you could mock the Write-Host function to just return a specific value or message when it is called with that argument.
That way, you can test that your script is correctly calling Write-Host with the expected argument, without actually affecting the outcome of the task.

A mock Write-Host in your test could look like:

Describe "Test Write-Host" {
    It "Should print out error if build requirement not found" {
        # Mock Write-Host to just return the message instead of printing it
        Mock Write-Host { return $args[0] }

        # Run your script or function that's supposed to call Write-Host with the failure message
        $result = Run-YourScript-Here

        # Check that Write-Host was called with the expected argument
        Assert-MockCalled Write-Host -ParameterFilter { $args[0] -eq "##vso[task.complete result=Failed;]DONE" }
    }
}

This way, you can verify that your script is correctly calling Write-Host with the failure message, without actually causing the task to fail.

You script becomes:

Import-Module Pester -Force

$config = [PesterConfiguration]::Default
$config.TestResult.Enabled = $true
$config.TestResult.OutputFormat = "NUnitXml"
$config.TestResult.OutputPath = "$env:SYSTEM_DEFAULTWORKINGDIRECTORY\TEST-results.xml"
$config.Run.Path = ".\my\path*"
$config.Run.PassThru = $true

# Mock Write-Host to just return the message instead of printing it
Mock Write-Host { return $args[0] }

$results = Invoke-Pester -Configuration $config

if($results.FailedCount -eq 0)
{
    Write-host "All tests passed"
    Write-Host "##vso[task.complete result=Succeeded;]DONE"
    exit 0
}

This will mock Write-Host for all the tests run by Invoke-Pester.
If you want to mock it for a specific test, you'll need to include the mock inside the Describe block of that test.