I have been trying to get the data lineage to populate with the simplest of queries on a unity enabled catalog with a unity enabled cluster. I am essentially running the example provided with more data to see how it works with various aggregates downstream.
However when I go to the lineage tab on the tables (which I have all privileges for the catalog and every schema/table underneath it) they are blank. There is history being collected and data is showing up in them.
I am running a cluster with 13.3 LTS and the following runtime settings enabled
CREATE TABLE IF NOT EXISTS
unity_metastore.lineage_test.menu (
recipe_id INT,
app string,
main string,
dessert string
)
USING DELTA LOCATION '[external location]';
INSERT INTO unity_metastore.lineage_test.menu
(recipe_id, app, main, dessert)
VALUES
(1,"Ceviche", "Tacos", "Flan"),
(2,"Tomato Soup", "Souffle", "Creme Brulee"),
(3,"Chips","Grilled Cheese","Cheesecake");
CREATE TABLE IF NOT EXISTS
unity_metastore.lineage_test.dinner (recipe_id INT,recipe_description string)
USING DELTA LOCATION '[external location]';
INSERT INTO unity_metastore.lineage_test.dinner (recipe_id,recipe_description)
SELECT
recipe_id, concat(app," + ", main," + ",dessert)
FROM
unity_metastore.lineage_test.menu;
I expect lineage data in diagram and lineage items in lineage tab page of databricks