How to make sure a file used by a steppable pipeline is closed even in case of an exception?

Question

This is a self-answered question. I'm going to add an answer only for PowerShell 7.3 or newer. Feel free to add answers for any previous PowerShell version.

---

I have a custom logging function that accepts pipeline input and uses a steppable pipeline object to correctly forward its input to Out-File. This answer provides background why a steppable pipeline is required in this case. Also see the blog post Mastering the (steppable) pipeline.

When any command of the pipeline of which my logging function is part of, throws an exception (aka script-terminating error) and the exception is caught, the file opened by Out-File doesn't get closed and any subsequent attempt to add something to the file (e. g. using Add-Content) fails with the error message:

The process cannot access the file 'C:\LogFile.log' because it is being used by another process.

Here is a reproducible example:

$LOG_FILE_PATH = Join-Path $PSScriptRoot LogFile.log

Function Test-SteppablePipeline {

    [CmdletBinding()]
    param (
        [Parameter( Mandatory, ValueFromPipeline )]
        [string] $InputObject
    )

    begin {
        Write-Host '[begin]'

        $steppablePipeline = {
            Out-File -LiteralPath $LOG_FILE_PATH
        }.GetSteppablePipeline( $MyInvocation.CommandOrigin )

        $steppablePipeline.Begin( $true )
    }

    process {
        Write-Host '[process]'
        
        $steppablePipeline.Process( $InputObject )

        $InputObject  # Forward to next command in pipeline of caller
    }

    end {
        Write-Host '[end]'

        $steppablePipeline.End()
    }
}

try {
    'foo', 'bar' | 
        ForEach-Object { throw 'my error' } |
        Test-SteppablePipeline
}
catch {
    '[catch]'
}

Write-Host '[add]'
Add-Content -LiteralPath $LOG_FILE_PATH -Value 'baz'

Output:

[begin]
[process]
[catch]
[add]
Add-Content: C:\Test.ps1:48
Line |
  48 |  Add-Content -LiteralPath $LOG_FILE_PATH -Value 'baz'
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The process cannot access the file 'C:\LogFile.log' because it is being used by another process.

As you can see by the [messages], the end block of Test-SteppablePipeline doesn't get called because of the exception, which propably is the reason why the file doesn't get closed.

Answer 1

A notable mention: This isn't an issue of just the steppable pipeline but an issue for any PowerShell function that obtains a handle and needs to ensure that said handle is properly disposed before exiting.

---

This is a big problem that got solved in PowerShell 7.3 with the clean block as zett42 points out in his helpful answer. For anyone wondering how this problem is solved in PowerShell 7.2 and previous versions, the answer is, it can't be solved with just PowerShell, the solution is to create a binary cmdlet that implements the IDisposable interface, the engine will do the rest (ensuring that .Dispose() is called invariantly, including CTRL + C or a terminating error).

A minimal repro of zett42's code using a binary cmdlet. Note that for the sake of demo, this is using Add-Type and inline C# for compiling at runtime, ideally, this cmdlet would be precompiled.

Add-Type '
using System;
using System.Management.Automation;

namespace Test
{
    [Cmdlet(VerbsDiagnostic.Test, "SteppablePipeline")]
    public sealed class TestSteppablePipelineCommand : PSCmdlet, IDisposable
    {
        private SteppablePipeline _pipe;

        [Parameter(Mandatory = true, ValueFromPipeline = true)]
        public string InputObject { get; set; }

        protected override void BeginProcessing()
        {
            WriteInformation("[begin]", null);
            _pipe = ScriptBlock.Create("Out-File -LiteralPath $LOG_FILE_PATH")
                .GetSteppablePipeline(MyInvocation.CommandOrigin);
            _pipe.Begin(true);
        }

        protected override void ProcessRecord()
        {
            WriteInformation("[process]", null);
            _pipe.Process(InputObject);
            WriteObject(InputObject, enumerateCollection: true);
        }

        protected override void EndProcessing()
        {
            WriteInformation("[end]", null);
            _pipe.End();
        }

        public void Dispose()
        {
            _pipe.Dispose();
        }
    }
}' -PassThru -IgnoreWarnings -WA 0 | Import-Module -Assembly { $_.Assembly }

$LOG_FILE_PATH = Join-Path $pwd.Path LogFile.log

try {
    0..10 |
        Test-SteppablePipeline -InformationAction Continue |
        ForEach-Object {
            if ($_ -eq 5) {
                throw 'my error'
            }
        }
}
catch {
    '[catch]'
}

Write-Host '[add]'
Add-Content -LiteralPath $LOG_FILE_PATH -Value 'baz'
Get-Content $LOG_FILE_PATH

Answer 2

The solution for PowerShell 7.3 or newer is trivial. PowerShell 7.3 adds a clean block to advanced functions. Citing from the documentation:

The clean block is a convenient way for users to clean up resources that span across the begin, process, and end blocks. It's semantically similar to a finally block that covers all other named blocks of a script function or a script cmdlet. Resource cleanup is enforced for the following scenarios:

1. when the pipeline execution finishes normally without terminating error
2. when the pipeline execution is interrupted due to terminating error
3. when the pipeline is halted by Select-Object -First
4. when the pipeline is being stopped by Ctrl+c or StopProcessing()

When using a steppable pipeline, you simply call the .Clean() method of the steppable pipeline object from within the clean block of the function that uses the steppable pipeline:

$LOG_FILE_PATH = Join-Path $PSScriptRoot LogFile.log

Function Test-SteppablePipeline {

    [CmdletBinding()]
    param (
        [Parameter( Mandatory, ValueFromPipeline )]
        [string] $InputObject
    )

    begin {
        $steppablePipeline = {
            Out-File -LiteralPath $LOG_FILE_PATH
        }.GetSteppablePipeline( $MyInvocation.CommandOrigin )

        $steppablePipeline.Begin( $true )
    }

    process {
        $steppablePipeline.Process( $InputObject )

        $InputObject  # Forward to next command in pipeline of caller
    }

    end {
        $steppablePipeline.End()
    }

    clean {
        # Makes sure the resources (e. g. files) allocated by the steppable pipeline 
        # are freed even in case of exceptions (script-terminating errors).
        $steppablePipeline.Clean()
    }
}