As a Redshift user can I query Databricks Delta Lake via Unity Catalog Hive Metastore as an external datasource?

162 views Asked by At

Summary:

what I am after is:

as a Redshift User... can I query Databricks Delta Lake tables via Unity Catalog as a federated query from Redshift?

I already know that

as a Databricks User... I can perform federated queries against Redshift.

[1] https://docs.databricks.com/en/query-federation/index.html

Detailed Question and Context

I'm seeking clarification of if anyone has had success with getting Redshift to query a Databricks Delta Lake via the Hive Meta Store API?

[2] https://docs.aws.amazon.com/redshift/latest/gsg/data-lake.html

...you use an external data catalog such as AWS Glue or your own Apache Hive metastore

[3] https://www.databricks.com/blog/extending-databricks-unity-catalog-open-apache-hive-metastore-api

June 9, 2023 ... we are excited to announce the preview of a Hive Metastore (HMS) interface for Databricks Unity Catalog, which allows any software compatible with Apache Hive to connect to Unity Catalog

  • Ideally I'd like Unity Catalog to be the arbiter of truth and access controls.
  • I have data in both Redshift and Databricks Delta Lake
  • I know that I can always LOAD a copy of data from Databricks into Redshift, I'd rather the data stay in place in this situation.

I think I can create a POC to test this out, I'd like to see if the internet could give me a faster response before sinking time into this. Also it helps others if this gets documented in public.

UPDATES:

  • 2023-11-20: As of this writing, the key feature of the HMS API exposed by Unity Catalog is still in private preview, so the answer is "no not yet". Will update as it gets to public preview and GA.
0

There are 0 answers