Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

Question

I am trying to create a metastore via the Terraform Azure databricks_metastore resource but I keep getting the error:

Error: cannot create metastore: Failed to check metastore quota limit for region westeurope. 

This is the exact code I am using to create the resource:

# Create the databricks metastore and assign it to the databricks workspace
resource "databricks_metastore" "this" {
  name          = "databricksmetastore"
  storage_root  = "abfss://[email protected]/"
  region        = "westeurope"
  owner         = ""
}

I have tried using both my Databricks account and a service principal application id, which are both in fact Account Admins, for the owner argument in the code block above to no avail.

Needless to say, there is no metastore in the region I am trying to create it in and if I use the GUI with my account I am able to successfully create and assign the metastore to the workspace.

Answer 1

I tried to trouble shoot the Issue Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region and I was able to provision requirement successfully.

It seems you're encountering an issue creating a Databricks metastore in the West Europe region using Terraform. The error message suggests a problem with checking the metastore quota limit, which could be due to a few different reasons.

1. Quota Limits: Your Azure subscription might have reached its quota limits for the resource you are trying to create.
2. Region Availability: There might be temporary unavailability or restrictions in the westeurope region.

This Issue can be overcome by making the changes in the type of data bricks account provisioned to Premium and by changing the region to any other available region which is readily available in the list of regions supported by Azure Databricks metastore. Here the region mentioned by you was allocated globally not a dedicated one associated with your Account subscription alone.

Here for our convenience, I tried using westus from the list of available region for the requirement you asked for.

My terraform configuration:

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}
provider "azurerm" {
  subscription_id = "enter your subscription id"
  features {}
}

data "azurerm_resource_group" "this" {
  name = "Vinay"
}

data "azurerm_databricks_workspace" "this" {
  name                = "Vinay-databricks"
  resource_group_name = "Vinay"
}

provider "databricks" {
  host = data.azurerm_databricks_workspace.this.workspace_url
}

resource "azurerm_databricks_access_connector" "unity" {
  name                = "vinaydatabricksmi"
  resource_group_name = data.azurerm_resource_group.this.name
  location            = data.azurerm_resource_group.this.location
  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_storage_account" "unity_catalog" {
  name                     = "thejadatabricksdemo"
  resource_group_name      = data.azurerm_resource_group.this.name
  location                 = data.azurerm_resource_group.this.location
  account_tier             = "Standard"
  account_replication_type = "GRS"
  is_hns_enabled           = true
}

resource "azurerm_storage_container" "unity_catalog" {
  name                  = "vinay-container"
  storage_account_name  = azurerm_storage_account.unity_catalog.name
  container_access_type = "private"
}

resource "azurerm_role_assignment" "example" {
  scope                = azurerm_storage_account.unity_catalog.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.unity.identity[0].principal_id
}

resource "databricks_metastore" "this" {
  name     = "primary"
  storage_root = format("abfss://%s@%s.dfs.core.windows.net/",
    azurerm_storage_container.unity_catalog.name,
  azurerm_storage_account.unity_catalog.name)
  force_destroy = true
  region        = "westus"
}

Output:

Terraform apply: