What is the most efficient trigger to use for an Azure Function to poll a container for new blobs every 5 minutes

Question

I need to create a function that polls a container every 5 minutes for new blobs, and then the data contained in each of these blobs will be forwarded on to an Event Hub to be picked up a SIEM system.

I wrote a PowerShell script which is based on a 5 minute timer trigger and loops through each blob and crafts a POST request which contains the data of the blob, sends this to the Event Hub and then finally deletes it (Script below).

However I'm not sure if there is a more efficient way to do this i.e. a blob storage trigger or event grid trigger. From what I understand, I think that these two types of triggers are basically run each time a new blob is created or a new blob event is generated respectively, and aren't able to batch process a number of blobs every 5 minutes.

Is it possible to achieve this using a different type of trigger and not creating a POST request per blob every time the function runs?

$resourceGroupName = "<resource_group_name>"
$storageAccountName = "<storage_account_name"
$storageAccountKey = (Get-AzKeyVaultSecret -VaultName "<vault_name>" -Name "<secret_name>").SecretValue | ConvertFrom-SecureString -AsPlaintext
$containerName = "testcontainer"
$EventHubsNamespace = "<eventhub_namespace_name>"
$EventHubsName = "<eventhub_name>"
$SharedAccessPolicyName = "<Shared_access_policy_name>"
$SharedAccessPolicyPrimaryKey = (Get-AzKeyVaultSecret -VaultName "<vault_name>" -Name "<secret_name>").SecretValue | ConvertFrom-SecureString -AsPlaintext

Connect-AzAccount -Identity

$storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
Write-Host "Connected to Storage Account: $($storageContext.Context.StorageAccountName)"

$containerContext =  Get-AzStorageContainer -Name $containerName -Context $storageContext
Write-Host "Connected to Container: $($containerContext.Name)"

$blobs = Get-AzStorageBlob -Container $containerContext.Name -Context $storageContext

foreach($blob in $blobs)
{
    Write-Host "Processing blob: $($blob.Name)"
    $blobContent = Get-AzStorageBlobContent -Blob $blob.Name -Container $containerContext.Name -Context $storageContext -Force

[Reflection.Assembly]::LoadWithPartialName("System.Web")| out-null
$URI = "$($EventHubsNamespace).servicebus.windows.net/$($EventHubsName)"
$Expires = ([DateTimeOffset]::Now.ToUnixTimeSeconds())+3600
$SignatureString = [System.Web.HttpUtility]::UrlEncode($URI)+ "`n" + [string]$Expires
$HMACSHA256 = New-Object System.Security.Cryptography.HMACSHA256
$HMACSHA256.key = [Text.Encoding]::ASCII.GetBytes($SharedAccessPolicyPrimaryKey)
$SignatureBytes = $HMACSHA256.ComputeHash([Text.Encoding]::ASCII.GetBytes($SignatureString))
$SignatureBase64 = [Convert]::ToBase64String($SignatureBytes)
$SASToken = "SharedAccessSignature sr=" + [System.Web.HttpUtility]::UrlEncode($URI) + "&sig=" + [System.Web.HttpUtility]::UrlEncode($SignatureBase64) + "&se=" + $Expires + "&skn=" + $SharedAccessPolicyName



$RestAPI = "https://$($EventHubsNamespace).servicebus.windows.net/$($EventHubsName)/messages"



$Headers = @{
        "Authorization"=$SASToken;
        "Content-Type"="application/json";
        }


$Body = $blobContent.ICloudBlob.DownloadText()

Invoke-RestMethod -Uri $RestAPI -Method "POST" -Headers $Headers -Body $Body   

Remove-AzStorageBlob -Container $containerContext.Name -Blob $blob.Name -Context $storageContext -Force

}

Answer 1

I do agree with @CSharpRocks that Logic apps if best way to do this and below is the design which does the same:

Firstly create a http trigger function like below:

Integrate your function app code.

Design in Logic app:

Choose the function in Azure functions Connector: