Log Pipeline Activity to Database after execution

Question

Currently, we are monitoring DataFlow Taks using Visual Link How to implement the log analysis and operations management to get desired Monitor pipeline runs and Monitor activity runs

Using ADF- Need Activity Runs information Inserted into a database table for each pipeline, enabling to create power bi report and share with monitoring team with my own charts and kpis

Kindly provide the Gitcode or steps to move the Monitor pipeline /activity runs into the database. Also, is there any quick way to export the result to excel

Answer 1

According to your description,i think the ADF azure monitor is your appropriate option. With Monitor, you can route diagnostic logs for analysis. Data Factory stores pipeline-run data for only 45 days.You have 3 options to store them:

1.Storage Account

2.Event hub

3.Log Analytics

You said your desired destination is database table.So, if the source data resides in blob storage then use ADF copy activity or Azure Blob Storage Trigger Function.If it is in the event hub,you could transfer them into db via ASA Job.