finding detailed error information in azure data factory pipeline

Question

I have a pipeline that started going very slow after a change. The issue was due to rows being skipped by the copy due to an error.

i was able to undo the change to a column formula and the pipeline runs fine now. the issue is that there was no indication of what was causing the issue.

Is there a way to get a detailed error message on why ADF is skipping rows?

Answer 1

1. Using Set Variable activity, we can capture the detailed error message of previous activity when error occurred.

   • Add output of copy activity on failure to Set Variable.

   

   • Create a variable on pipeline level to store the message.

   

   • When you select Set variable activity, in variables drop down, it gives the list of variables available. Select the variable created to store error message.

   

   • Instead of hardcoding the message in Value, we will capture from previous activity output. For that we can add dynamic content for the value.

   • Add the expression @activity('Copy data1').Error.Message in the dynamic content. Error.Message is used to capture error message.

   

   

   • Run the pipeline and you can see the Set variable activity is triggered when copy activity is failed.

   • Click on the output of set variable and you can see the detailed error message of the error occurred.

   

2. You can log the skipped rows or skipped files by enabling Session log in copy activity and store these log files in a Storage location.

   When you select "Warning" as logging level, it will log skipped files and skipped rows only.

   Once the activity is ran, log file is available in the linked path for download.

Reference : Session log in copy activity