Azure Data Factory Managed Airflow integration with Purview to get Lineage

127 views Asked by At

I am trying to use Azure Data Factory managed Airflow for my use cases. I am trying to use Azure Purview as data governance tool and want to use airflow to emit lineage to it. I am trying to follow the Microsoft Document here. However, what i am stuck at is after i create event hub how do i configure the .yml file as Azure does it for me and i have no access to it. I am also not able to create the OPENLINEAGE_URL correctly pointing to Eventhub and it is throwing me 401(assuming i am not providing SAS correctly). This is the URL i am providing:
https://{event-hub-namespace}.servicebus.windows.net/{event-hub}/messages

1

There are 1 answers

1
1ridesteel On

i am also on the same process but have not reached the expected result yet.

Regarding the configuration of the .yml file there is a suggestion at the link below: https://learn.microsoft.com/en-us/answers/questions/1405026/azure-data-factory-managed-airflow-integration-wit

mentioning the following:

Did you try to create a new Variable and provide the YML contents?

  1. Create a new variable.
  2. In the "Key" field, enter "openlineage_config_yaml".
  3. In the "Value" field, enter the contents of your .yml file.
  4. Click on the "Save" button to save the variable.
  5. Restart the Airflow scheduler to apply the changes.