Progress bar is not showing correctly in PowerShell 7 in ForEach -Parallel

Question

this show 0%:

this show 4%:

the realted script:

$iWrapper = [hashtable]::Synchronized(@{ i = 0 })
$srcfile = "C:\Users\wnune\OneDrive\Escritorio\imagenes\cardlist.txt"
$urls = Get-Content $srcfile
$lines = 0
switch -File $srcfile { default { ++$lines } }
Write-Host "Total Urls to process: $lines "
Write-Progress -Activity "Downloading files" -Status "In progress" -PercentComplete $i;
$urls | ForEach-Object -Parallel {
    try {
        $url = $_
        $filename = Split-Path $url -Leaf
        $destination = "C:\Users\wnune\OneDrive\Escritorio\imagenes\$filename"
        $ProgressPreference = 'SilentlyContinue'
        $response = Invoke-WebRequest -Uri $url -ErrorAction SilentlyContinue
        if ($response.StatusCode -ne 200) {
            Write-Warning "============================================="
            Write-Warning "Url $url return Error. "
            continue
        }
        if (Test-Path $destination) {
            Write-Warning "============================================="
            Write-Warning "File Exist in Destination: $filename "
            continue
        }
        $job = Start-BitsTransfer -Source $url -Destination $destination -Asynchronous
        while (($job | Get-BitsTransfer).JobState -eq "Transferring" -or ($job | Get-BitsTransfer).JobState -eq "Connecting")
        {
            Start-Sleep -m 250
        }
        Switch(($job | Get-BitsTransfer).JobState)
        {
            "Transferred" {
                Complete-BitsTransfer -BitsJob $job
            }
            "Error" {
                $job | Format-List
            }
        }
    }
    catch 
    {
        Write-Warning "============================================="
        Write-Warning "There was an error Downloading"
        Write-Warning "url:         $url"
        Write-Warning "file:        $filename"
        Write-Warning "Exception Message:"
        Write-Warning "$($_.Exception.Message)"
    }
    $j = ++($using:iWrapper).i
    $k = $using:lines
    $percent = [int](100 * $j / $k)
    Write-Host "PercentCalculated: $percent"
    Write-Host "Progress bar not Show the %"
    Write-Progress -Activity "Downloading files " -Status " In progress $percent" -PercentComplete $percent
}
Write-Progress -Activity "Downloading files" -Status "Completed" -Completed

If I am passing in -PercentComplete $percent which is an integer why does the progress bar not receive it correctly?

I have verified that the script and the environment are correctly configured but I cannot validate because the progress bar is not seen correctly.

Answer 1

Note:

• A potential future enhancement has been proposed in GitHub issue #13433, suggesting adding parameter(s) such as -ShowProgressBar to ForEach-Object -Parallel so that it would automatically show a progress bar based on how many parallel threads have completed so far.

• A simpler solution than the ones below can be found in miljbee's helpful answer, which simply delegates updating the progress play to an additional, non-parallel ForEach-Object call, which receives the outputs from the parallel threads as they are being emitted.

---

Leaving the discussion about whether Start-BitsTransfer alone is sufficient aside:

At least as of PowerShell v7.3.1, it seemingly is possible to call Write-Progress from inside threads created by ForEach-Object -Parallel, based on a running counter of how many threads have exited so far.

However, there are two challenges:

• You cannot directly update a counter variable in the caller's runspace (thread), you can only refer to an object that is an instance of a .NET reference type in the caller's runspace...

• ...and modifying such an object, e.g. a hashtable must be done in a thread-safe manner, such as via System.Threading.Monitor.

Note that I don't know whether calling Write-Progress from different threads is officially supported, but it seems to work in practice, at least when the call is made in a thread-safe manner, as below.

• Bug alert, as of PowerShell 7.3.1: Irrespective of whether you use ForEach-Object -Parallel or not, If you call Write-Progress too quickly in succession, only the first in such a sequence of calls takes effect:
  • See GitHub issue #18848
  • As a workaround, the code below inserts a Start-Sleep -Milliseconds 200 call after each Write-Progress call.
    • Not only should this not be necessary, it slows down overall execution, because threads then take longer to exit, which affects not only a given thread, but overall execution time, because it delays when threads "give up their slot" in the context of thread throttling (5 threads are allowed to run concurrently by default; use -ThrottleLimit to change that.

A simple proof of concept:

# Sample pipeline input
$urls = 1..100 | ForEach-Object { "foo$_" }

# Helper hashtable to keep a running count of completed threads.
$completedCount = @{ Value = 0 }

$urls | 
  ForEach-Object -parallel { # Process the input objects in parallel threads.

    # Simulate thread activity of varying duration.
    Start-Sleep -Milliseconds (Get-Random -Min 0 -max 3000)
    # Produce output.
    $_

    # Update the count of completed threads in a thread-safe manner
    # and update the progress display.
    [System.Threading.Monitor]::Enter($using:completedCount) # lock access
      ($using:completedCount).Value++
      # Calculate the percentage completed.
      [int] $percentComplete = (($using:completedCount).Value / ($using:urls).Count) * 100
      # Update the progress display, *before* releasing the lock.
      Write-Progress -Activity Test -Status "$percentComplete% complete" -PercentComplete $percentComplete
      # !! Workaround for the bug above - should *not* be needed.
      Start-Sleep -Milliseconds 200
    [System.Threading.Monitor]::Exit($using:completedCount) # release lock

  }

---

An alternative approach in which the calling thread centrally tracks the progress of all parallel threads:

Doing so requires adding the -AsJob switch to ForEach-Object -Parallel, which, instead of the synchronous execution that happens by default, starts a (thread-based) background job, and returns a [System.Management.Automation.PSTasks.PSTaskJob] instance that represents all parallel threads as PowerShell (thread) jobs in the .ChildJobs property.

A simple proof of concept:

# Sample pipeline input
$urls = 1..100 | ForEach-Object { "foo$_" }

Write-Progress -Activity "Downloading files" -Status "Initializing..."

# Launch the parallel threads *as a background (thread) job*.
$job = 
  $urls |
    ForEach-Object -AsJob -Parallel {
      # Simulate thread activity of varying duration.
      Start-Sleep -Milliseconds (Get-Random -Min 0 -max 3000)
      $_ # Sample output: pass the URL through
    }

# Monitor and report the progress of the thread job's 
# child jobs, each of which tracks a parallel thread.
do {

  # Sleep a bit to allow the threads to run - adjust as desired.
  Start-Sleep -Seconds 1 

  # Determine how many jobs have completed so far.
  $completedJobsCount  = 
     $job.ChildJobs.Where({ $_.State -notin 'NotStarted', 'Running' }).Count

  # Relay any pending output from the child jobs.
  $job | Receive-Job

  # Update the progress display.
  [int] $percent = ($completedJobsCount / $job.ChildJobs.Count) * 100
  Write-Progress -Activity "Downloading files" -Status "$percent% complete" -PercentComplete $percent

} while ($completedJobsCount -lt $job.ChildJobs.Count)

# Clean up the job.
$job | Remove-Job

While this is more work and less efficient due to the polling loop, it has two advantages:

• The script blocks running in the parallel threads need not be burdened with progress-reporting code.

• The polling loop affords the opportunity to perform other foreground activity while the parallel threads are running in the background.

• The bug discussed above needn't be worked around, assuming your Start-Sleep interval in the polling loop is at least 200 msecs.

Answer 2

I propose a much easier solution: piping the results to another foreach-object which isn't parallel to display the progress-bar. have a look at the test code, it will be clearer:

# Generate data
$list = 1..100

# Processing data
$list |
    ForEach-Object -parallel {
        # Simulate thread activity of varying duration.
        Start-Sleep -Milliseconds (Get-Random -Min 0 -max 300)
        # Produce output
        $_
    } |
    # Piping the results to a queued foreach-object which 
    # will write progress as results are delivered
    ForEach-Object -Begin { $received = 0 } -Process { 
        # incrementing result count
        $received += 1
        # Calculate the percentage completed.
        [int] $percentComplete = ($received / $list.Count) * 100
        # update progress bar
        Write-Progress -Activity Test -Status "$percentComplete% complete" -PercentComplete $percentComplete
        # returning the result
        $_
    }