Using databricks workspace in the same configuration as the databricks provider

Question

I'm having some trouble getting the azurerm & databricks provider to work together.

With the azurerm provider, setup my workspace

resource "azurerm_databricks_workspace" "ws" {
  name                        = var.workspace_name
  resource_group_name         = azurerm_resource_group.rg.name
  location                    = azurerm_resource_group.rg.location
  sku                         = "premium"
  managed_resource_group_name = "${azurerm_resource_group.rg.name}-mng-rg"
  custom_parameters {
    virtual_network_id  = data.azurerm_virtual_network.vnet.id
    public_subnet_name  = var.public_subnet
    private_subnet_name = var.private_subnet
  }
}

No matter how I structure this, I can't say seem to get the azurerm_databricks_workspace.ws.id to work in the provider statement for databricks in the the same configuration. If it did work, the above workspace would be defined in the same configuration and I'd have a provider statement that looks like this:

provider "databricks" {
  azure_workspace_resource_id = azurerm_databricks_workspace.ws.id
}

Error:

I have my ARM_* environment variables set to identify as a Service Principal with Contributor on the subscription.

I've tried in the same configuration & in a module and consuming outputs. The only way I can get it to work is by running one configuration for the workspace and a second configuration to consume the workspace.

This is super suboptimal in that I have a fair amount of repeating values across those configurations and it would be ideal just to have one.

Has anyone been able to do this? Thank you :)

Answer 1

I've had the exact same issue with a not working databricks provider because I was working with modules. I separated the databricks infra (Azure) with databricks application (databricks provider).

In my databricks module I added the following code at the top, otherwise it would use my azure setup:

terraform {
  required_providers {
    databricks = {
      source = "databrickslabs/databricks"
      version = "0.3.1"
    }
  }
}

In my normal provider setup I have the following settings for databricks:

provider "databricks" {
  azure_workspace_resource_id = module.databricks_infra.databricks_workspace_id
  azure_client_id             = var.ARM_CLIENT_ID
  azure_client_secret         = var.ARM_CLIENT_SECRET
  azure_tenant_id             = var.ARM_TENANT_ID
}

And of course I have the azure one. Let me know if it worked :)

Answer 2

From the error message, it looks like Authentication is not configured for provider could you please configure it through the one of following options mentioned above.

For more details, refer Databricks provider - Authentication.

For passing the custom_parameters, you may checkout the SO thread which addressing the similar issue.

In case if you need more help on this issue, I would suggest to open an issue here: https://github.com/terraform-providers/terraform-provider-azurerm/issues

Answer 3

If you experience technical difficulties with rolling out resources in this example, please make sure that environment variables don't conflict with other provider block attributes. When in doubt, please run TF_LOG=DEBUG terraform apply to enable debug mode through the TF_LOG environment variable. Look specifically for Explicit and implicit attributes lines, that should indicate authentication attributes used. The other common reason for technical difficulties might be related to missing alias attribute in provider "databricks" {} blocks or provider attribute in resource "databricks_..." {} blocks. Please make sure to read alias: Multiple Provider Configurations documentation article.