I'm retrieving Azure Data Factory logs for analysis using Powershell.
I am successfully retrieving the top level log (the pipeline) and the log nested inside that (activities) and writing to a text file.
However I'm having issues flattening the activities file, which consists of a mix of flat records and fields containing json
Here is my abridged script. The issue is with the last cmdlet call. I need to work out how to flatten it out to what I need
$DateFrom = (new-object System.DateTime 2018, 07, 01)
$DateTo = Get-Date
$Pipeline="MyPipeline"
$Outputfile="C:\SRC\ADF\$Pipeline.TXT"
$OutputSubfile="C:\SRC\ADF\$Pipeline.Sub.TXT"
$DFname ="MyDataFactory"
$RG="MyRG"
$TenantId="a16xxxx-xxx-xxx"
$Subscription="d8xxx-xxx-xxx"
$Credential = Get-Credential
Connect-AzureRmAccount `
-TenantId $TenantId `
-Subscription $Subscription `
-Credential $Credential
$oADFLog = Get-AzureRmDataFactoryV2PipelineRun `
-ResourceGroupName $RG `
-DataFactoryName $DFname `
-LastUpdatedAfter $DateFrom `
-LastUpdatedBefore $DateTo `
-PipelineName $Pipeline
# This is the pipeline log - it works as required
$oADFLog | Export-Csv -Path $Outputfile -Delimiter "`t" -NoTypeInformation
# Delete the subtask file
Remove-Item -Path $oADFSubLog -Force -Recurse -ErrorAction Ignore
Foreach ($PipelineRun IN $oADFLog)
{
# For each parent run ID, check the child tasks
# File results in thispart need to be cleaned up
$oADFSubLog = Get-AzureRmDataFactoryV2ActivityRun `
-PipelineRunId $PipelineRun.RunId `
-ResourceGroupName $PipelineRun.ResourceGroupName `
-DataFactoryName $PipelineRun.DataFactoryName `
-RunStartedAfter $DateFrom `
-RunStartedBefore $DateTo
# This is the activity log - it has nested data types and is ugly
# I need to flatten the Json inside the message
$oADFSubLog | Export-Csv -Append -Path $OutputSubfile -Delimiter "`t" -NoTypeInformation
}
Given $oADFSubLog
in the above script, I've discovered that I can pull out some piece that I need like this:
(ConvertFrom-Json -InputObject $oADFSubLog[0].Input.ToString()).packageLocation
This pulls the property that I need out of the Json
But I'm not sure how to easily push that out to a file in conjunction with the other flat attributes
I've tried this, which is really just a stab in the dark
$oADFSubLog | Select-Object -Property ActivityName,@(ConvertFrom-Json -InputObject $oADFSubLog[0].Input.ToString()).packageLocation
But I get
Select-Object : Cannot convert System.Management.Automation.PSObject to one of the following types {System.String, System.Management.Automation.ScriptBlock}.
I've seen a few examples of custom cmdlets and scripts that can be added but I don't want to go there just yet - I just want to understand how to do this.