Retry BITS file transfer for failed downloads with Powershell

Question

I'm doing a BITS transfer of daily imagery from a web server and I keep getting random drops during the transfer.

As it's cycling through the downloads I get the occasional "The connection was closed prematurely" or "An error occurred in the secure channel support". There are about 180 images in each folder and this happens for maybe 5-10% of them. I need to retry the download for those that didn't complete.

My code to do so follows - my imperfect work-around is to run the loop twice but I'm hoping to find a better solution.

# Set the URL where the images are located
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'

# Set the local path where the images will be stored
$path = 'C:\images\Wind_Waves\latest\'

# Create a list of all assets returned from $url
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | Where-Object{ $_.tagName -eq 'A' -and $_.href.ToLower().EndsWith("jpg") }

# Create a list of all href items from the table & call it $images
$images = $table.href 

# Enumerate all of the images - for troubleshooting purposes - can be removed
$images

# Check to make sure there are images available for download - arbitrarily picked more than 2 $images
if($images.count -gt 2){

    # Delete all of the files in the "latest" folder
    Remove-Item ($path + "*.*") -Force
# For loop to check to see if we already have the image and, if not, download it
ForEach ($image in $images)
{
if(![System.IO.File]::Exists($path + $image)){
    Write-Output "Downloading: " $image
    Start-BitsTransfer -Source ($url + $image) -Destination $path -TransferType Download -RetryInterval 60
    Start-Sleep 2
    }
}
Get-BitsTransfer | Where-Object {$_.JobState -eq "Transferred"} | Complete-BitsTransfer
} else {
Write-Output "No images to download"}

Answer 1

I don't see any error handling in your code to resume/retry/restart on fail.

1. Meaning why is there no try/catch in the loop or the Get?
2. If the Get is on per download job in the loop, why is it outside the loop?

Download is the default for TransferType, so no need to specify, it normally will generate an error if you do.

So, something like this. I did test this, but never got a fail. Yet, I have a very high-speed speed internet connection. If you are doing this inside an enterprise, edge devices (filters, proxies, could also be slowing things down, potentially forcing timeouts.)

$url  = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object{
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)

<#
Check to make sure there are images available for download - arbitrarily 
picked more than 2 $images
#>
if($images.count -gt 2)
{
    Remove-Item ($path + '*.*') -Force
    ForEach ($image in $images)
    {
        Try
        {
            Write-Verbose -Message "Downloading: $image" -Verbose

            if(![System.IO.File]::Exists($path + $image))
            {
                $StartBitsTransferSplat = @{
                    Source              = ($url + $image) 
                    Destination         = $path 
                    RetryInterval       = 60
                }
                Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                Start-Sleep 2
            }

            Get-BitsTransfer | 
            Where-Object {$PSItem.JobState -eq 'Transferred'} | 
            Complete-BitsTransfer
        }
        Catch
        {
            $PSItem.Exception.Message
            Write-Warning -Message "Download of $image not complete or failed. Attempting a resume/retry" -Verbose
            Get-BitsTransfer -Name $image | Resume-BitsTransfer

        }
    }
} 
else 
{
    Write-Warning -Message 'No images to download'
    $PSItem.Exception.Message
}

See the help files

Resume-BitsTransfer Module: bitstransfer Resumes a BITS transfer job.

# Example 1: Resume all BITS transfer jobs owned by the current user
Get-BitsTransfer | Resume-BitsTransfer

# Example 2: Resume a new BITS transfer job that was initially suspended
$Bits = Start-BitsTransfer -DisplayName "MyJob" -Suspended
Add-BitsTransfer -BitsJob $Bits  -ClientFileName C:\myFile -ServerFileName http://www.SomeSiteName.com/file1
Resume-BitsTransfer -BitsJob $Bits -Asynchronous

# Example 3: Resume the BITS transfer by the specified display name
Get-BitsTransfer -Name "TestJob01" | Resume-BitsTransfer

Answer 2

Here is a combination of the code that postanote provided and a Do-While loop to retry the download up to 5x if an error is thrown.

$url  = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object{
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>

($images = $table.href)

 <# Check to make sure there are images available for download - arbitrarily 
    picked more than 2 $images #>

    if($images.count -gt 2)
    {
        Remove-Item ($path + '*.*') -Force
        ForEach ($image in $images)
        {
            # Create a Do-While loop to retry downloads up to 5 times if they fail
            $Stoploop = $false
            [int]$Retrycount = "0"
            do{
            Try
            {
                Write-Verbose -Message "Downloading: $image" -Verbose
    
                if(![System.IO.File]::Exists($path + $image))
                {
                    $StartBitsTransferSplat = @{
                        Source              = ($url + $image) 
                        Destination         = $path 
                        RetryInterval       = 60
                    }
                    Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                    Start-Sleep 10
                    $Stoploop = $true
                }
    
                Get-BitsTransfer | 
                Where-Object {$PSItem.JobState -eq 'Transferred'} | 
                Complete-BitsTransfer
            }
            Catch
            {
                if ($Retrycount -gt 5){
                $PSItem.Exception.Message
                Write-Warning -Message "Download of $image not complete or failed." -Verbose
                $Stoploop = $true
                }
                else {
                Write-Host "Could not download the image, retrying..."
                Start-Sleep 10
                $Retrycount = $Retrycount + 1
                }
            }
        }
        While ($Stoploop -eq $false)
        }
    }
    else 
    {
        Write-Warning -Message 'No images to download'
        $PSItem.Exception.Message
    }

Answer 3

Here's a somewhat modified version of the above code. It appears the BITS transfer job object goes away when the error occurs, so there is no use in trying to find/resume that job. Instead, I wrapped the entire Try-Catch block in a while loop with an exit when the file is downloaded.

$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
$MaxRetries = 3     # Initialize the maximum number of retry attempts.

# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | 
Where-Object {
    $_.tagName -eq 'A' -and 
    $_.href.ToLower().EndsWith('jpg') 
}

<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)

<#
Check to make sure there are images available for download - arbitrarily 
picked more than 2 $images
#>
if ($images.count -gt 2) {
    Remove-Item ($path + '*.*') -Force
    ForEach ($image in $images) {
        # Due to occasional failures to transfer, wrap the BITS transfer in a while loop
        # re-initialize the exit counter for each new image
        $retryCount = 0
        while ($retryCount -le $MaxRetries){
            Try {
                Write-Verbose -Message "Downloading: $image" -Verbose

                if (![System.IO.File]::Exists($path + $image)) {
                    $StartBitsTransferSplat = @{
                        Source        = ($url + $image) 
                        Destination   = $path 
                        RetryInterval = 60
                    }
                    Start-BitsTransfer @StartBitsTransferSplat -ErrorAction Stop
                    Start-Sleep 2
                }

                # To get here, the transfer must have finished, so set the counter
                # greater than the max value to exit the loop
                $retryCount = $MaxRetries + 1
            } # End Try block

            Catch {
                $PSItem.Exception.Message
                $retryCount += 1
                Write-Warning -Message "Download of $image not complete or failed. Attempting retry #: $retryCount" -Verbose
            } # End Catch Block
            
        } # End While loop for retries

    } # End of loop over images

} # End of test for new images
else {
    Write-Warning -Message 'No images to download'
    $PSItem.Exception.Message
} # End of result for no new images