Databricks Unity Catalog - Error: cannot create metastore data access

Question

I'm in the progress of enabling Databricks Unity Catalog and encountered a problem with the databricks_metastore_data_access Terraform resource:

resource "databricks_metastore_data_access" "this" {
  provider = databricks.account-level

  metastore_id  = databricks_metastore.this.id
  name          = "mi-access-databricks-metastore"
  owner         = "admin-group"
  force_destroy = true
  is_default    = true

  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.this.id
  }
}

The Terraform apply run will always fail with following error: Error: cannot create metastore data access
Unfortunately I get no more error details than the message from above.

The funny thing is that all other Unity Catalog related resource were created successfully and looking at the Account Console I can see that the metastore was assigned to my Databricks workspace.

So my question: Do you know for what the metastore data access resource is used for and do I even need it? If yes, how can I fix this issue?

Answer 1

I did not find a direct solution to this problem, however we don't need this resource anymore at all. Turns out you can also create a Metastore without the underlying storage infrastructure. As we have environment separated external storage locations anyway, the Metastore storage account is obsolete for us. This means the only two resources we need in order to configure the Metastore is to just use the databricks_metastore and the databricks_metastore_assignment resource. This important fact is still missing in the official documents and should be updated.

Answer 2

The Terraform apply run will always fail with following error: Error: cannot create metastore data access:

The error message cannot create metastore typically indicates an issue with the Databricks provider configuration. In Terraform code, the databricks_metastore resource is attempting to create a Metastore in Databricks workspace, and it relies on the Databricks provider configuration to authenticate and communicate with Databricks account.

I also got the same error as you when I tried to create a meta store with an incorrect provider configuration.

To resolve the issue, pass the correct Databricks workspace details in the provider and check the Access Permissions. Follow the MS Doc to learn about the data metastore requirement.

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}
provider "azurerm" {
  subscription_id = "fnfnf-4d98-95c5-fngng"
  features {}
}

data "azurerm_resource_group" "this" {
  name = "Venkat"
}

data "azurerm_databricks_workspace" "this" {
  name                = "venkat-databricks"
  resource_group_name = "Venkat"
}

provider "databricks" {
  host = data.azurerm_databricks_workspace.this.workspace_url
}

resource "azurerm_databricks_access_connector" "unity" {
  name                = "venkatdatabricksmi"
  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location
  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_storage_account" "unity_catalog" {
  name                     = "thejadatabricksdemo"
  resource_group_name      = data.azurerm_resource_group.this.name
  location                 = data.azurerm_resource_group.this.location
  account_tier             = "Standard"
  account_replication_type = "GRS"
  is_hns_enabled           = true
}

resource "azurerm_storage_container" "unity_catalog" {
  name                  = "venkat-container"
  storage_account_name  = azurerm_storage_account.unity_catalog.name
  container_access_type = "private"
}

resource "azurerm_role_assignment" "example" {
  scope                = azurerm_storage_account.unity_catalog.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.unity.identity[0].principal_id
}

resource "databricks_metastore" "this" {
  name     = "primary"
  storage_root = format("abfss://%s@%s.dfs.core.windows.net/",
    azurerm_storage_container.unity_catalog.name,
  azurerm_storage_account.unity_catalog.name)
  force_destroy = true
  region        = "westus"
}

Terraform apply:

Once ran the above code, the data bricks Metastore has been created.