Powershell: Copy new entries from file after the last run

Question

I'm trying to create a powershell script to copy only the new entries that match a pattern (can be one or multiple lines) from one file to another, every time the script is run. The source file is updated randomly by an application but the request is to copy the last entries every hour.

The solution I'm working on is to take the last entry from previous run stored in a file and the compare with the last entry from the file, and if these don't match, then start copying the new lines after that one. That is the part when I'm stuck, I can't figure it how to indicate that, instead I'm copying the whole content every time.

This is what I got so far:

Write-Host "Declaring the output log file ..... " -ForegroundColor Yellow
$destinationfile = 'C:\file\out\output.log'


Write-Host "Getting the last line of the source file ..... " -ForegroundColor Yellow
$sourcefile = 'C:\app\blade\inputlogs.log'
$sourcefilelastline = Get-Content $originfile | Select-Object -last 1
$sourcefilelastline


Write-Host "Getting the last line of the destination file ..... " -ForegroundColor Yellow
$destinationfilelastline = Get-Content $destinationfile | Select-Object -last 1
$destinationfilelastline


if ($sourcefilelastline -eq $destinationfilelastline){
    Write-Host "Skipping the process ..... " -ForegroundColor Yellow
}
else{
    Write-Host "Reading the source log file and updating destination file  ..... " -ForegroundColor Yellow
    $sourcefilecontent = Get-Content -Path $sourcefile | Where-Object { $_ -ne '' } | Select-String -Pattern 'error' -CaseSensitive -SimpleMatch
    $sourcefilecontent | Add-Content $destinationfile
}

Any ideas on how to get this done ? Thanks.

Answer 1

Get-content have a switch 'tail' which lets you read the last rows from a file. Per Microsoft own words:

The Tail parameter gets the last line of the file. This method is faster than retrieving all the lines in a variable and using the [-1] index notation.

You can use it in your case start from the bottom line and go up until they match.

<# The Function.
.SYNOPSIS
 Recive X last number of lines from a file

.DESCRIPTION
Using the "Tail" parameter of get-content we can get the X number of lines from a file.
This should dramatically improve performance instead of reading the entire file.

.PARAMETER FilePath
Mandatory parameter.
Path to the file to get the X number of lines from.

.PARAMETER LastLines
Optional parameter.
How many lines to read from the end of the file.
Default value = 1

.EXAMPLE
Get-FileLastContent -FilePath "C:\Windows\System32\drivers\etc\hosts" -LastLines 1

.NOTES
General notes
#>

function Get-FileLastContent {
    param (
        [Parameter(
            Position = 0,
            Mandatory = $true,
            HelpMessage = 'Path to File')]
        [ValidateScript({ Test-Path -Path $_ })]
        [string]
        $FilePath,

        [Parameter(
            Position = 1,
            HelpMessage = 'Number of lines lines to check')]
        [ValidateNotNullOrEmpty()]
        [int]
        $LastLines = 1
    )



    $FileContent = Get-Content -Path $FilePath -Tail $LastLines
    Return $FileContent 
}

## Inisialization of Variables, as sometimes re-running the script is using values from last run.
$LastLines = 1
$destinationfilelastline = ''
$sourcefilelastline = ''



##Declaring the Destnation File
Write-Host -Object  "Declaring the output log file ..... " -ForegroundColor Yellow
$destinationfile = 'C:\file\out\output.log'

## Using the function to recive it's last line
Write-Host -Object "Getting the last line of the destination file ..... " -ForegroundColor Yellow
$destinationfilelastline = Get-FileLastContent -FilePath $destinationfile
$destinationfilelastline


## Declaring the source file and using the function to get it's last line
Write-Host -Object "Getting the last line of the source file ..... " -ForegroundColor Yellow
$sourcefile = 'C:\app\blade\inputlogs.log'
$sourcefilelastline = Get-FileLastContent -FilePath $sourcefile -LastLines 1
Write-Host -Object $sourcefilelastline

## if source file is empty, empty destination file as well
if ($sourcefilelastline.Length -eq 0) {
    Write-Host -Object "Soruce file is empty, clearing destination" -ForegroundColor Yellow
    Set-Content -Path $destinationfile -Value "" -Force -Encoding UTF8 -NoNewline
}

## If source file is not empty, but not matching the destination file
elseif (($sourcefilelastline -ne $destinationfilelastline)) {
    Write-Host -Object "Reading the source log file and updating destination file  ..... " -ForegroundColor Yellow
    ## if destination file is not empty, loop in the source file bottom-up until it is finding a line matching the destinaion file
    if ($destinationfilelastline.Length -gt 0) {
        while (($sourcefilelastline[0] -ne $destinationfilelastline) -and ($LastLines -le $sourcefile.Length)) {
            $LastLines = $LastLines + 1
            $sourcefilelastline = Get-FileLastContent -FilePath $sourcefile -LastLines $LastLines
        }
        ## If found a match in source file compared to the destination file
        if (($sourcefilelastline[0] -eq $destinationfilelastline)) {
            ## Prepare sourcefilelastline variable for export, skip the first result as it is already in the destination file (they match).
            $sourcefilelastline = $sourcefilelastline | Select-Object -Skip 1 | Where-Object { $_ -ne '' } | Select-String -Pattern 'error' -CaseSensitive -SimpleMatch
            # adding new line at the start
            $sourcefilelastline[0] = "`n" + $sourcefilelastline[0]
            ## Export the sourcefilelastline to the destination file
            $sourcefilelastline | Out-File -FilePath $destinationfile -Force -Encoding UTF8 -Append
        }
        else {
            ## it means that the sourcefile was overwriteen since last check with complete new data
            ## Overwrite destination file with the last line of source file
            $sourcefilelastline[($sourcefilelastline.Length-1)] | Out-File -FilePath $destinationfile -Force -Encoding UTF8
        }
    }
    #If no match found in sourcefile compare to the destination file
    
    # if destination file is empty, copy the last line of the source file to the destination file.
    else {
        $sourcefilelastline | Out-File -FilePath $destinationfile -Force -Encoding UTF8
    }
}
## Skip if source and destination are equal
else {
    Write-Host -Object "Skipping the process ..... " -ForegroundColor Yellow
}

Answer 2

This is a little bit of an experiment, but seems to work well.

The function Read-LastLinesOfTextFile:

1. Accepts an $AppName parameter which is used to define a path in the Registry to store values - in this case, the position that the last read of the file ended at.
   • The idea here is that you could have multiple scripts calling this function, but because you use a unique name per script, there will not be any conflicts between them.
2. Accepts a $TextFile parameter which is the path of the file that needs to be read.
3. Returns the lines read as an array of strings.
4. Uses seek to set the read position of file stream that is used to read the file.
5. After reading from the file stream, retrieves the new position and saves it to registry where it will be used the next time the function called.

function Read-LastLinesOfTextFile {
    param(
        [Parameter(Mandatory = $true, Position = 0)]
        [string]$AppName,
        [Parameter(Mandatory = $true, Position = 1)]
        [string]$TextFile
    )
    $RegistryPath = "HKCU:\Software\$AppName"               # Registry key path for storing registry value
    if (-not (Test-Path "$RegistryPath")) {                 # If key path does not exists
        $null = New-Item -Path $RegistryPath -Force         #   Then create registry key
    }
    # Check if the registry value for LastPosition does NOT exists - https://stackoverflow.com/a/43921551/4190564
    if ( (Get-ItemProperty "$RegistryPath").PSObject.Properties.Name -notcontains "LastPosition" ) {
        # Create LastPosition value
        $null = New-ItemProperty -Path $RegistryPath -Name "LastPosition" -Value 0 -PropertyType DWORD -Force
    }
    # Save registry value LastPosition to $LastPosition
    $LastPosition = Get-ItemPropertyValue -Path $RegistryPath -Name "LastPosition"
    $CurrentFileSize = (Get-Item $TextFile).Length          # Get current file size
    if ($CurrentFileSize -lt $LastPosition) {               # If the file is smaller than it used to be
        $LastPosition = 0                                   #   Then assume it was deleted and now has all new data.
    } elseif ($CurrentFileSize -lt $LastPosition) {
        return @()
    }
    # Open file stream
    try { $FileStream = New-Object System.IO.FileStream($TextFile, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read) }
    catch { return @() }
    # Open stream reader
    try { $StreamReader = New-Object System.IO.StreamReader($FileStream) }
    catch {
        $FileStream.Close()                                 # Close FileStream
        return @()
    }
    $Return = @()                                           # Define default return value of empty array
    try {
        if ($LastPosition) {                                # If LastPosition anything other than 0
            # Seek last position
            $null = $FileStream.Seek($LastPosition, [System.IO.SeekOrigin]::Begin)
        }
        $Text = $StreamReader.ReadToEnd()                   # Read to the end of the file
        # Update the registry with the new last position
        Set-ItemProperty -Path $RegistryPath -Name "LastPosition" -Value $FileStream.Position
        $Return = $Text -split "\r?\n"                      # Split string into lines. https://stackoverflow.com/a/76831908/4190564
    }
    finally {
        $StreamReader.Close()                               # Close StreamReader
        $FileStream.Close()                                 # Close FileStream
    }
    return $Return
}

To call this function, use a line similar to the following, only make sure to replace MyLogReader with a name unique to your script so you don't have conflicts with other script using the same function. It looks like you are doing a case sensitive comparison to "error", so I used -cmatch in this example. Change that to -match if you want case insensitive.

Read-LastLinesOfTextFile 'MyLogReader' "$PSScriptRoot\MyLogFile.LOG" | Where-Object { $_ -cmatch 'error' } | Add-Content $destinationfile