Azure functions read, process and save file on arrival in blob

Question

I have a use case where I have to process files when they arrive in blob container.

The files are in a wiresharkproprietary format and need to be processed with specific program (tshark).

My goal is to execute the shell command that converts files to csv:

tshark -r <input_filename> <some other params> > <output_filename>.csv

and then save output csv files to another blob container.

I know that I can set a blob trigger in azure functions to trigger the process, but I'm not sure how to save the output file to blob and make sure that tshark is available in the runtime.

Another question is: maybe there is a better way to do this rather than azure functions?

Answer 1

I managed to achieve my goal by doing the following:

1. Push docker image with all dependencies installed to Azure Container Registry
2. Create Event Grid Subscription to system topic (blob created in storage account with additional filters like container and file extension). This populates the storage queue with messages.
3. Create Container App Job with storage queue trigger that uses the docker image and consumes messages in the queue. For storage account access, I put the Container App Environment in my VNET.

Answer 2

there is a better way to do this rather than azure functions?

Yes, you can use Azure Logic apps and Azure Automation Accounts .

Below is the way how I got expected results:

This calls the automation account.

In Automation account you can add tshark module and use them.

Then you can edit your runbook and include parameters in it get the name of file. In run book get blob using blob name got from from Logic apps.

Here just printing name, you can write commands to get blob from blobstorage and use tsharks with it. You can also create temparory(SO-Thread where I used Temperoroy file) file in Runbook.

Now use the blobname get the blob and then use tshark and then upload to another storage container.