Does Auto Loader process the file again when the file gets appended or overwritten?

246 views Asked by At

I have the case where data owner has enabled some extra columns in D365 this week. for example I had a table with 5 columns last week but starting from this week I have extra 6th column in that specific table.

Problem is that I dont know how the Azure Synapse Link for Dataverse ( append only mode) acts in that case. Main question is will it resend all the files ( with updated 6th column) again that has already been processed by autoloader or it will overwrite old files.

If it will overwrite old file then I have a probem that autolaoder will never ever read those files again sice Im using file notification mode.

My main question is, is there a way that I can tell autolaoder that if there has been some modification (overwrite or append) on old files, it will still pick them up and act on it?

I saw this documentation but did not really help to understand

0

There are 0 answers