ETL / ELT pipelines - Metainformation about the pipeline

98 views Asked by At

how do you add metainformation about the used ETL / ELT code (and version of this ELT code) to the produced sink files / tables?

Do u consider it as required to have information like "PipelineID" or "DataProductionTime" in the targetfolder?

1

There are 1 answers

1
Rakesh Govindula On BEST ANSWER

how do you add metainformation about the used ETL / ELT code (and version of this ELT code) to the produced sink files / tables?

You can do it using the pipeline dynamic content and additional column in copy activity.

This is my source file:

enter image description here

In source of copy activity use additional column.

enter image description here

In ADF you can find meta information about the pipeline in System variables of dynamic content.

enter image description here

Target file in sink folder:

enter image description here

Do u consider it as required to have information like "PipelineID" or "DataProductionTime" in the targetfolder?

All this depends on your requirement like if you want to know by which pipeline you are getting data you can use this.